Content recommendation system with weighted metadata annotations

ABSTRACT

Systems and methods are described for improving content classifications and metadata for a plurality of content items. The content items may comprise various forms of media content. A content recommendation processing system may automatically annotate and/or classify content items with a plurality of metadata tag information describing content items. The disclosed system and methods may also determine and/or assign applicability scores indicating a level or degree of suitability for annotating a content item with particular metadata tag information, and may utilize applicability scores to provide content to calibrate metadata information for a plurality of content items, and further to provide content recommendations to users.

BACKGROUND

Current methods of predicting user entertainment and/or contentfavorites can be complicated, time-consuming and generate inaccurateresults. Given the increasing volume and availability of programmingcontent, it has become remarkably difficult to efficiently process andidentify meaningful content for a consumer. The flexibility offered bycontent recommendation and classification systems can be beneficial, assuch, there remains an ever-present need for improved and simplifiedways of processing and identifying content similarities to alleviateprocessing burdens and maximize computing resources for contentrecommendation systems.

SUMMARY

The following summary is for illustrative purposes only, and is notintended to limit or constrain the detailed description. The followingsummary merely presents various described aspects in a simplified formas a prelude to the more detailed description provided below.

Features herein relate to an improved content recommendation processingcomputer and methods, which may be used as a stand-alone recommendationsystem or comprise a portion of a content recommendation system.According to one aspect of the disclosure herein, an improved contentrecommendation processing computer and/or system may be used to generatemetadata for one or more content items made available to users of acontent service. The content recommendation processing computer mayautomatically annotate content items with a plurality of metadata fieldsand corresponding tags (e.g., metadata tag values) describing thecontent items. Additionally or alternatively, the content recommendationprocessing computer may determine and/or apply an applicability score toa tag in a metadata field of a content item to indicate a level/degreeof similarity between the content item and a plurality of programs thathave been assigned the same metadata tag. According to another aspect ofthe disclosure herein, the content recommendation processing computermay utilize determined metadata annotations to calibrate and/or toconfirm the accuracy of existing metadata for a plurality of contentitems, and further may utilize applicability scores to recommend contentto users.

The summary here is not an exhaustive listing of the novel featuresdescribed herein, and are not limiting of the claims. These and otherfeatures are described in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood with regard to the followingdescription, claims, and drawings. The present disclosure is illustratedby way of example, and not limited by, the accompanying figures in whichlike numerals indicate similar elements.

FIG. 1 shows an example communication network on which various featuresdescribed herein may be used.

FIG. 2 shows an example computing device that can be used to implementany of the methods, servers, entities, and computing devices describedherein.

FIG. 3A shows an exemplary system for assigning metadata andcorresponding applicability scores for a content recommendationprocessing system according to one or more illustrative aspects of thedisclosure.

FIG. 3B shows an exemplary diagram of a similarity matrix according toone or more illustrative aspects of the disclosure.

FIG. 4A shows an exemplary flowchart of a process for applying metadatato content according to one or more illustrative aspects of thedisclosure

FIG. 4B shows an exemplary diagram associated with a process forapplying metadata to content according to one or more illustrativeaspects of the disclosure

FIG. 5 shows an exemplary flowchart of a process for generating and/orassigning metadata and corresponding applicability scores according toone or more illustrative aspects of the disclosure.

FIG. 6 shows an exemplary flowchart of a process for calibratingmetadata and applicability scores for content according to one or moreillustrative aspects of the disclosure.

FIG. 7 shows an exemplary flowchart of a process for generating contentrecommendations based on determined applicability scores according toone or more illustrative aspects of the disclosure.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

FIG. 1 shows an example communication network 100 on which many of thevarious features described herein may be implemented. The communicationnetwork 100 may be any type of information distribution network, such assatellite, telephone, cellular, wireless, etc. One example may be anoptical fiber network, a coaxial cable network, or a hybrid fiber/coaxdistribution network. Such communication networks 100 use a series ofinterconnected communication links 101 (e.g., coaxial cables, opticalfibers, wireless, etc.) to connect the various premises 102 (e.g.,businesses, homes, consumer dwellings, etc.) to a local office orheadend 103. The local office 103 may transmit downstream informationsignals onto the communication links 101, and each of the variouspremises 102 may have a receiver used to receive and process thosesignals.

There may be one communication link originating from the local office103, and it may be split a number of times to distribute the signal tothe various premises 102 in the vicinity (which may be many miles) ofthe local office 103. The communication links 101 may include componentsnot illustrated, such as splitters, filters, amplifiers, etc. to helpconvey the signal clearly, but in general each split introduces a bit ofsignal degradation. Portions of the communication links 101 may also beimplemented with fiber-optic cable, while other portions may beimplemented with coaxial cable, other lines, or wireless communicationpaths.

The local office 103 may include an interface 104, such as a terminationsystem (TS) interface 104. More specifically, the interface 104 may be acable modem termination system (CMTS), which may be a computing deviceconfigured to manage communications between devices on the network oflinks 101 and backend devices such as servers 105-107 (to be discussedfurther below). The interface 104 may be as specified in a standard,such as the Data Over Cable Service Interface Specification (DOCSIS)standard, published by Cable Television Laboratories, Inc. (a.k.a.CableLabs), or it may be a similar or modified device instead. Theinterface 104 may be configured to place data on one or more downstreamfrequencies to be received by modems at the various premises 102, and toreceive upstream communications from those modems on one or moreupstream frequencies.

The local office 103 may also include one or more network interfaces108, which can permit the local office 103 to communicate with variousother external networks 109. These external networks 109 may include,for example, networks of Internet devices, telephone networks, cellulartelephone networks, fiber optic networks, local wireless networks (e.g.,WiMAX), satellite networks, and any other desired network, and thenetwork interface 108 may include the corresponding circuitry needed tocommunicate on the external networks 109, and to other devices on thenetwork such as a cellular telephone network and its corresponding cellphones.

As noted above, the local office 103 may include a variety of computingdevices 105-107, such as servers, that may be configured to performvarious functions. For example, the local office 103 may include a pushnotification computing device 105. The push notification device 105 maygenerate push notifications to deliver data and/or commands to thevarious premises 102 in the network (or more specifically, to thedevices in the various premises 102 that are configured to detect suchnotifications). The local office 103 may also include a content servercomputing device 106. The content device 106 may be one or morecomputing devices that are configured to provide content to users attheir premises. This content may be, for example, video on demandmovies, television programs, songs, text listings, etc. The contentserver computing device 106 may include software to validate useridentities and entitlements, to locate and retrieve requested content,to encrypt the content, and to initiate delivery (e.g., streaming) ofthe content to the requesting user(s) and/or device(s). Indeed, any ofthe hardware elements described herein may be implemented as softwarerunning on a computing device.

The local office 103 may also include one or more application servercomputing devices 107. The application server 107 may be a computingdevice configured to offer any desired service, and may run variouslanguages and operating systems (e.g., servlets and JSP pages running onTomcat/MySQL, OSX, BSD, Ubuntu, Red Hat, HTML5, JavaScript, AJAX andCOMET). For example, an application server may be responsible forcollecting television program listings information and generating a datadownload for electronic program guide listings. The application server107 may be responsible for monitoring user viewing habits and collectingthat information for use in selecting advertisements. The applicationserver may also be responsible for formatting and insertingadvertisements in a video stream being transmitted to the variouspremises 102. Although shown separately, one of ordinary skill in theart will appreciate that the push notification device 105, contentserver computing device 106, and the application server 107 may becombined. Further, here the push notification device 105, the contentserver computing device 106, and the application server 107 are showngenerally, and it will be understood that they may each contain memorystoring computer executable instructions to cause a processor to performsteps described herein and/or memory for storing data.

The example premise 102 a, such as a home, may include an interface 120.The interface 120 may include any communication circuitry needed toallow a device to communicate on one or more communication links 101with other devices in the network. For example, the interface 120 mayinclude the modem 110, which may include transmitters and receivers usedto communicate on the communication links 101 and with the local office103. The modem 110 may be, for example, a coaxial cable modem (forcoaxial cable lines 101), a fiber interface node (for fiber optic lines101), twisted-pair telephone modem, cellular telephone transceiver,satellite transceiver, local Wi-Fi router or access point, or any otherdesired modem device. Also, although only one modem is shown in FIG. 1,a plurality of modems operating in parallel may be implemented withinthe interface 120. Further, the interface 120 may include a gatewayinterface device 111. The modem 110 may be connected to, or be a partof, the gateway interface device 111. The gateway interface device 111may be a computing device that communicates with the modem(s) 110 toallow one or more other devices in the premises 102 a, to communicatewith the local office 103 and other devices beyond the local office 103.The gateway interface device 111 may be a set-top box (STB), digitalvideo recorder (DVR), computer server, or any other desired computingdevice. The gateway interface device 111 may also include (not shown)local network interfaces to provide communication signals to requestingentities/devices in the premises 102 a, such as the display devices 112(e.g., televisions), STB and/or DVR 113, the personal computers 114, thelaptop computers 115, the wireless devices 116 (e.g., wireless routers,wireless laptops, notebooks, tablets and netbooks, cordless phones(e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones,mobile televisions, personal digital assistants (PDA), etc.), thelandline phones 117 (e.g. Voice over Internet Protocol—VoIP phones), thetablet computing devices 118, the mobile phones 119, and any otherdesired devices. Examples of the local network interfaces includeMultimedia over Coax Alliance (MoCA) interfaces, Ethernet interfaces,universal serial bus (USB) interfaces, wireless interfaces (e.g., IEEE802.11, IEEE 802.15), analog twisted pair interfaces, Bluetoothinterfaces, and others.

FIG. 2 shows general hardware elements that may be used to implement anyof the various computing devices discussed herein. The computing device200 may include one or more processors 201, which may executeinstructions to perform any of the features described herein. Theinstructions may be stored in any type of computer-readable medium ormemory, to configure the operation of the processor 201. For example,instructions may be stored in a read-only memory (ROM) 202, the randomaccess memory (RAM) 203, the removable media 204, such as a UniversalSerial Bus (USB) drive, compact disk (CD) or digital versatile disk(DVD), floppy disk drive, or any other desired storage medium.Instructions may also be stored in an attached (or internal) hard drive205. The computing device 200 may also include a security processor (notshown), which may execute instructions of a one or more computerprograms to monitor the processes executing on the processor 201 and anyprocess that requests access to any hardware and/or software componentsof the computing device 200 (e.g., ROM 202, RAM 203, the removable media204, the hard drive 205, the device controller 207, a network circuit209, the GPS 211, etc.). The computing device 200 may include one ormore output devices, such as the display 206 (e.g., an externaltelevision), and may include one or more output device controllers 207,such as a video processor. There may also be one or more user inputdevices 208, such as a remote control, keyboard, mouse, touch screen,microphone, etc. The computing device 200 may also include one or morenetwork interfaces, such as the network circuit 209 (e.g., a networkcard) to communicate with an external network 210. The network circuit209 may be a wired interface, wireless interface, or a combination ofthe two. In some embodiments, the network circuit 209 may include amodem (e.g., a cable modem), and the external network 210 may includethe communication links 101 discussed above, the external network 109,an in-home network, a provider's wireless, coaxial, fiber, or hybridfiber/coaxial distribution system (e.g., a DOCSIS network), or any otherdesired network. Additionally, the device may include alocation-detecting device, such as a global positioning system (GPS)microprocessor 211, which may be configured to receive and processglobal positioning signals and determine, with possible assistance froman external server and antenna, a geographic position of the device.

The FIG. 2 example is a hardware configuration, although the illustratedcomponents may be implemented as software as well. Modifications may bemade to add, remove, combine, divide, etc. components of the computingdevice 200 as desired. Additionally, the components illustrated may beimplemented using basic computing devices and components, and the samecomponents (e.g., processor 201, ROM storage 202, display 206, etc.) maybe used to implement any of the other computing devices and componentsdescribed herein. For example, the various components herein may beimplemented using computing devices having components such as aprocessor executing computer-executable instructions stored on acomputer-readable medium, as shown in FIG. 2. Some or all of theentities described herein may be software based, and may co-exist in acommon physical platform (e.g., a requesting entity can be a separatesoftware process and program from a dependent entity, both of which maybe executed as software on a common computing device).

One or more aspects of the disclosure may be embodied in acomputer-usable data and/or computer-executable instructions, such as inone or more program modules, executed by one or more computers or otherdevices. Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types when executed by a processor ina computer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects of the disclosure, and such datastructures are contemplated within the scope of computer executableinstructions and computer-usable data described herein. The variouscomputing devices, servers and hardware described herein may beimplemented using software running on another computing device.

There are numerous ways in which a content recommendations system mayrecommend content to a user. In some instances, a recommendation systemmay generate/determine and store metadata for content items, and mayrecommend a content item to a user based on the stored metadata. Inother instances, a recommendation system may recommend content based onanalyzing which users have consumed content items, and analyzing theparticular content items that have been consumed. For example, a contentrecommendation system may determine user-based content recommendation bymatching a first user to one or more other users based on determiningwhether content consumed by the first user matches content itemsconsumed by one or more other users. The recommendation system maydetermine whether content items consumed by the first user matchescontent items consumed by one or more other users by analyzing storedmetadata information for the respective content items. If a match isfound, the content recommendation system may recommend, to a first user,content consumed by and/or recommended to the matching user.

As another example, the content recommendation system may determineitem-based content recommendations by matching a particular content itemto one or more other content items and comparing users that haveconsumed a particular content item to other users that have consumed oneor more other content items. In this example, the recommendation systemmay be configured to determine a similarity between two content items.The similarity of a first content item (p) to a second content item (q)(denoted as sim(p, q)) may indicate how much more than randomly expecteda user will like the first content item, given that the user liked thesecond content item. This similarity determination may be expressed asthe ratio of the number of users that like a first item and a seconditem to the expected number of users that like the first item and thesecond item. Accordingly, a similarity determination much greater than 1may indicate that two content items are very similar, while a similaritydetermination less than 1 may indicate the two content items aredissimilar. As will be discussed in more detail below, the contentrecommendation processing system may determine a similarity between twocontent items in a variety of other manners, such as by comparingmetadata, analyzing visual/textual information of the content items, andthe like.

For example, the similarity value between a first content item (p) and asecond content item (q) may be expressed as:

ratio(p,q)=P(q|p)/P(q)

where:

-   -   “P(q|p)” comprises the probability that a user will consume        content item (q) given that the user has previously consumed        content item (p); and        where:    -   “P(q)” comprises the probability that a user has consumed        content item (q).

In some instances, the content recommendation processing system maydetermine P(q|p) by determining a fraction/proportion of the totalnumber of users (e.g., the plurality of users that subscribe to and/orare members of the content service) that have consumed content item (p)and content item (q). In this example, the content recommendationprocessing system may determine P(q) by determining thefraction/proportion of the total number of users that have consumedcontent item (q). Accordingly, a similarity (e.g., similarity value)between the first content item and the second content item may bedetermined based on the ratio of the fraction of total users that haveconsumed the first and second content items to the fraction of usersthat have consumed the second content item. Given that a similarityvalue indicating the similarity between two content items may beexpressed as a ratio, the magnitude of the similarity value may indicatethe level of similarity or dissimilarity between the content items, andthus a determined similarity value may vary between a wide range ofvalues. However, in some instances, a system administrator may preferthat determined similarity values fall within a predetermined scale ofvalues, such as between 0 and 1. Accordingly, in some embodiments, thecontent recommendation processing system may be configured to adjustand/or normalize determined similarity values.

As described above, to determine which content to recommend to aparticular user, content recommendations systems may analyze themetadata of a content item to determine and compare the content item toother content items. Additionally or alternatively, a content servicemay provide metadata for content to users such that the user maydistinguish between the various content items made available forconsumption. Typically, analyzing the subject matter and characteristicsof content may require a human viewer to consume the content andmanually assign labels (e.g., metadata) describing the content. Forexample, a human editor may watch a movie, and may assign the movie ametadata field (e.g., genre, actor, theme, keyword, etc.). If the editorassigns the movie a genre metadata field, the editor may also assign oneor more corresponding metadata tag values (e.g., horror, romance,comedy, etc.) to the genre metadata field describing various genreattributes that apply to and/or characterize the subject matter of themovie.

The entire set of metadata fields and corresponding metadata tag valuesused to characterize content may be very large, and as such, the manualassignment of metadata to content may be very tedious, incomplete, andsubjective. Furthermore, the use of human editors to assign metadata tocontent may be error-prone, largely because manually assigning metadatato content can be a binary endeavor. Either the content item comprises acertain characteristic and is assigned the metadata, or the content itemdoes not comprise the characteristic and the metadata is not assigned bythe editor. However, real-world descriptions and characterizations ofcontent may be more nuanced than the binary determination describedabove, and more importantly, each tag value applied to a metadata fieldmay not equally characterize or represent the subject matter of thecontent item.

For example, the genre metadata field for a movie may be assigned aromance tag and a comedy tag to describe the genre and subject matter ofthe movie. However, simply labeling the movie as a romance movie andlabeling the movie as a comedy movie fails to indicate to what extentthe movie embodies the characteristics of and/or is similar to othercomedy and romance movies, respectively. For example, the entire plot ofthe movie may focus on a love story between characters, and the moviemay include brief moments of levity intermingled within the storylinefor the movie. Such a movie may be more similar to a prototypicalromance movie than a comedy movie. By contrast, if the entire plot ofthe movie focused on the crazy antics of a group of friends during abachelor party where the groom briefly rekindles a relationship with apast girlfriend, such a movie may be more similar to a prototypicalcomedy movie than a romance movie. Accordingly, assigning a weighting orscore to the various values/tags of a metadata field for a content itemmay provide the user with additional information to proportionatelyassess the subject matter of the content item. As will be explained inmore detail below, in some aspects of the present disclosure, thecontent recommendation processing system may be configured to generatean applicability score that is appropriate when annotating a metadatafield (e.g., genre) of a content item with a particular metadata tagvalue (e.g., horror).

The applicability score determined for a particular metadata tag valueand content item indicates a level or degree of similarity between thecontent item and a plurality of programs that have also been associatedand/or annotated with the particular metadata tag value. For example,when determining whether to annotate the genre metadata field of aparticular movie with a horror metadata tag, an applicability scoredetermined for the horror metadata tag and the movie indicates a levelof similarity between the particular movie and other movies that havebeen annotated with a horror metadata tag. Accordingly, theapplicability score may represent a degree of suitability for annotatinga content item with a particular metadata tag value.

As discussed above, each content item made available to users by acontent service may be initially associated with a limited amount ofmetadata, such as the metadata fields and corresponding tags that weremanually applied to the content item by a system administrator. However,given the large number of metadata fields and corresponding tagsavailable to characterize the subject matter of content, a variety ofthese metadata fields and corresponding tags may not be assigned to eachcontent item available for consumption by users. For example, thecontent service may use 391 different metadata tag values (e.g., horror,comedy, fantasy, etc.) to characterize the genre of a movie. However, asdiscussed above, an editor may only apply/assign a subset of thoseavailable metadata tag values to a genre metadata field of each moviemade available to users. As described herein, in some aspects of thepresent disclosure, a content recommendation processing system may beconfigured to generate and assign metadata (e.g., metadata fields andcorresponding tags) to content items based on determined applicabilityscores. As will be explained in more detail below, the contentrecommendation processing system may utilize existing metadata fieldsand corresponding tag values for a first set of content items togenerate and assign metadata for another set of content items.

In some aspects of the present disclosure, the content recommendationprocessing system may be configured to calibrate and/or quality check anexisting metadata tag value for a content item in view of anapplicability score determined for the metadata tag and content item. Asnoted above, the applicability score for a particular metadata tag valueand content item may indicate the degree of similarity between thecontent item and other content items that have also been annotated withthe particular metadata tag. Thus, the appropriateness of an existingmetadata tag value assigned to a content item may be assessed based onthe applicability score determined for the existing metadata tag value.In particular, the content recommendation processing system maydetermine whether to remove or maintain the existing metadata tag valuefor the content item by comparing the determined applicability score toa threshold value (e.g., threshold applicability score).

In other aspects of the present disclosure, the content recommendationprocessing system may utilize the applicability score for a particularmetadata tag value and content item when determining whether to applyand/or annotate a metadata field of the content item with the particularmetadata tag. As noted above, an applicability score may indicate howapplicable or suitable it would be to annotate a content item with aparticular metadata tag value. The content recommendation processingsystem may determine whether to apply or annotate a metadata tag valueto the content item by comparing the applicability score determined fora metadata tag value and content item to a threshold value. For example,when determining whether to assign a horror metadata tag value to thegenre metadata field of a movie, the content recommendation processingsystem may compare the applicability score for the horror metadata tagvalue and movie to a threshold applicability score. Various thresholdvalues may be utilized by the content recommendation processing systemwhen assessing the applicability/suitability of a metadata tagannotation.

As will be explained in more detail below, the content recommendationprocessing system may use applicability scores to determine whether toapply available metadata tag values to content items that have not beenannotated with certain metadata tag values or any metadata tag values,such as newly-released movies and/or classic movies that are being addedto a content server for the first time by the content service. Assessingmetadata fields for content items based on determined applicabilityscores may allow the content recommendation processing system to moreefficiently process and identify meaningful content for a user and togenerate/improve metadata for content items such that the user isprovided with more precise content recommendations and programminginformation for content items, while also reducing the amount ofcomputing resources required by the content recommendation processingcomputer to classify/annotate a large volume of content items. In someaspects of the present disclosure, applicability scores may be used toaffect which content items are recommended to a user, and/or may affectthe presentation of content items and/or programming information to auser on a content service application or user interface (e.g., anelectronic program guide).

FIG. 3A shows an exemplary system for generating metadata tag values andcorresponding applicability scores for a content recommendationprocessing system. The system 300 includes a content recommendationprocessing computer 301, a tag applicability module 302, a weightingdatabase 303, a content recommendation module 304, a program similaritymodule 305, a tag database 306, one or more content servers 310, aninput device 315, and one or more client devices 320. One or moreportions of system 300 may reside in one or more computing devicesresiding in various locations. The one or more computing devicescomprising the content recommendation processing system may beimplemented as a computing device using the structure shown in FIG. 2,and may be configured to respond to user requests for content, such asstreaming video or audio. For example, the content recommendationprocessing computer 301 may be configured to receive user requests forcontent and/or user feedback from the input device 315 and/or the clientdevice 320. The input device 315 may comprise a remote control,keyboard, mouse, touch screen, microphone, and/or other suitable meansfor receiving user input. In some aspects of the present disclosure, theinput device 315 may incorporate aspects and/or features of input device208 described above.

As shown in FIG. 3A, in the content recommendation processing computer301 may be operatively connected to and/or in communication with the tagdatabase 306, which stores a plurality of metadata tag values that maybe assigned to one or more metadata fields of a content item. In someaspects of the present disclosure, a content item may comprise aplurality of metadata fields and metadata tag values used to describecharacteristics of the content item. As noted above, metadata tag valuesmay be used to describe a particular characteristic of a content itemand/or a portion of a content item. Each metadata tag value may beassociated with and/or assigned to a particular metadata field. Examplesof metadata fields and corresponding metadata tag values for videocontent may include genre (e.g., animated, family entertainment,children entertainment, fantasy, horror, action, fantasy, comedy,adventure, documentary, drama, romance, adaption, foreign, sciencefiction, paranormal, etc.); theme (e.g., imaginary kingdom, daringrescues, evil aliens, save the world, space travel, fantasy lands,heroic mission, curses and spells, fall in love, save the day, chasing adream, state affairs, class differences, etc.); tone (e.g., fanciful,creepy, deadly, gruesome, humorous, rousing, stylized, witty, quirky,satirical, deadpan, clever, sarcastic, etc.); keyword (e.g., friendship,pop-culture, hero, partner, fantasy-world, danger, UFO, alien, disaster,chaos, etc.). Various other metadata fields and associated metadata tagvalues may be stored within the tag database 306 and utilized by thecontent recommendation processing system without departing from thescope of the present disclosure. In some embodiments, metadata fieldsand metadata tag values may also be used to describe other types ofcontent, such as audio content and the like.

In some aspects of the present disclosure, the tag database may bepopulated with metadata fields and metadata tag values used todescribe/characterize content stored in the content server 310. Thefields and tag values may be imported from another computing device ormay include the various fields and tag values that have been used by thecontent service over time to describe content made available to users.In some embodiments, a user may populate the tag database withadditional metadata tag values. The client device 320 may be configuredto execute a content service application or other suitable interface(e.g., an electronic program guide) such that a user may browse andselect content items for consumption. In some embodiments client device320 may incorporate one or more aspects and/or features of the computingdevice 200 described above in reference to FIG. 2. The client device 320may be further configured to receive user input data indicatingcharacteristics of a content item that may be used to generate and/orassign a metadata tag value to a content item. The client device may beconfigured to generate a metadata tag value utilizing one or morecharacteristics of a content item and/or a portion of a content itemprovided by the user, and the client device may be further configured totransmit the generated metadata tag value to tag database 306.Accordingly, a portion of the metadata tag values stored in tag database306 may comprise metadata tag values created by system administrators,while other metadata tag values in tag database 306 may comprisemetadata tag values created by users.

The content server 310 may comprise one or more computing devicesconfigured to provide content items to users at their premises (e.g., atthe client device 320). As discussed above, content items may be, forexample, video on demand movies, television programs, songs, textlistings, etc. In some embodiments, the client device 310 mayincorporate one or more features and/or aspects of the content server106 described above with respect to FIG. 1. Although not shown in FIG.3A, the client device 320 may be operatively connected to the contentserver 310, and may further be configured to request and/or receivecontent items stored at the content server 310. Content consumptionhistories and viewing habits for a plurality of users of the contentservice may be stored at the content server 310, and/or another suitablecomputing device operatively connected to the content server 310. Asusers consume and/or reserve content items for consumption, the contentserver 310 and/or other suitable computing device may monitor and storeeach user's viewing and consumption habits. In some embodiments, theconsumption history data for a user may comprise video-on-demand (VOD)viewing habits (e.g., where a user may actively decide which contentitems to consume), linear content viewing habits (e.g., where a user iswatching content items broadcast by a television network), and/or acombination of both.

In some aspects of the present disclosure, a portion of the contentitems stored at the content server 310 may comprise existing metadatatag information describing different characteristics of each contentitem. For example, in some instances, a content service may utilizesystem administrators and/or other editors to review and annotate aportion of the content items that are stored in the content server 310.Over time, metadata information may be assigned to a plurality ofcontent items stored at the content server 310. However, as discussedabove, the process of manually reviewing and annotating a largerepository of content items in view of a large set of available metadatatag values may be arduous, time consuming, and incomplete. Additionally,the manual assignment of metadata tag values may typically prioritizethose content items that are most popular and/or that have been consumedby a large number of users. Furthermore, it may be largely unreasonableand/or inefficient to expect editors to watch millions of content itemsin order to homogenously (and accurately) assign metadata tag values toone or more metadata fields/attributes for each content item, and inparticular, assigning metadata tag values to sparsely-consumed contentitems.

The program similarity module 305 may be used to alleviate the burden ofmanually creating metadata tag values for content items stored in acontent server, and may comprise one or more computing devicesconfigured to generate and/or store a usage-based similarity matrixbetween content items (e.g., content items stored at the content server310). As discussed above, a computing device (e.g., a collaborativefiltering recommendation computer) may determine a value (e.g., sim(p,q)) indicating a similarity between a first content item (p) and asecond content item (q). In some instances, the program similaritymodule 305 may be configured to determine a similarity value between twocontent items. Additionally or alternatively, the program similaritymodule 305 may be configured to retrieve from a computing device thesimilarity value between two content items. In some instances, thesimilarity value between a first content item and a second content itemmay be expressed as the ratio of a number of users that like the firstitem and the second item to the expected number of users that like thefirst item and the second item.

In other instances, the similarity value between a first content itemand a second content item may be expressed as the ratio of the fractionof total users that have consumed the first and second content items tothe fraction of total users that have consumed the second content item.The program similarity module 305 or another suitable computing device,may be configured to retrieve and process the content consumptionhistory for a plurality of users to determine the similarity valuebetween content items. The program similarity module 305 may beconfigured to transmit similarity data to another computing device uponrequest. For example, the program similarity module 305 may beconfigured to determine the respective similarity values between a firstcontent item and a plurality of other content items (e.g., content itemsstored in the content server 310), and transmit similarity dataindicating the determined similarity values to another computing device.In some embodiments, the content recommendation processing computer 301may be configured to generate and/or store a usage-based similaritymatrix between content items, as described above.

FIG. 3B shows an example usage-based similarity matrix that may begenerated by the program similarity module 305 for a number of contentitems (“n”) stored in the content server 310. As shown in FIG. 3B, eachcolumn and each row in the matrix may represent a content item stored inthe content server, such that the matrix includes “n” columns and “n”rows. Each entry in the matrix may indicate a similarity value betweentwo content items. The program similarity module 305 may determine asimilarity between a first content item and each remaining content itemin the content server. For example, as shown in FIG. 3B, the programsimilarity module 305 may determine a similarity value between item 1and item 2 (i.e., 1.5), between item 1 and item 3 (i.e., 0.7), and soforth until a similarity value has been determined between item 1 andthe n^(th) content item. The program similarity module 305 may continueto populate the matrix by determining similarity values for eachpotential pairing of content items. As shown in FIG. 3B, the programsimilarity module 305 may not determine a similarity value between thesame content item (i.e., sim(item 1, item 1)). Accordingly, the programsimilarity module 305 may determine a similarity between any two contentitems using the similarity matrix. Given that user content preferencesand the number of content items stored on the content server 310 mayvary, program similarity module 305 may be configured to dynamicallyand/or periodically update the information stored in the similaritymatrix.

The usage-based similarity data generated by and/or stored at theprogram similarity module 305 indicates a similarity between contentitems, and as will be explained in more detail below with respect toFIGS. 4A-B, the program similarity module 305 may determine similaritiesin metadata tag values between content items. In some embodiments, thetag applicability module 302 may request from program similarity module305 the similarity values for a pair of content items. Any number ofmethods may be utilized by the program similarity module 305 and/or thecontent recommendation processing system 300 to determine a level ofsimilarity (e.g., similarity values) between content items withoutdeparting from the scope of the present disclosure. The programsimilarity module 305 may generate the similarity matrix based on theobtained data indicating a level of similarity between content items.

The tag applicability module 302 may be configured to generate anapplicability score for a metadata tag value that has been and/or can beassigned to a content item. As shown in FIG. 3A, the tag applicabilitymodule 302 may be in communication with the program similarity module305, the tag database 306, and the content server 310. In some aspectsof the present disclosure, for any given content item (P) stored in thecontent server 310 and metadata tag value (T) stored at the tag database306, the tag applicability module 302 may utilize similarity data storedat program similarity module 305 to generate an applicability scoreindicating the suitability of assigning “t” to a metadata field of P. Insome embodiments, the tag applicability module 302 may be configured togenerate applicability scores within a predetermined threshold scale ofvalues. For example, the tag applicability module 302 may generateapplicability scores between 0 and 1 for a metadata tag value. Asanother example, the tag applicability module 302 may generateapplicability scores between 0 and 100. Various threshold applicabilityscore scales may be utilized by the tag applicability module 302 withoutdeparting from the scope of the present disclosure.

FIG. 4A shows an example method for determining an applicability scorefor a metadata tag value and content item according to one or moreaspects described herein. The steps shown in FIG. 4A may be performed bya content recommendation processing system, for example, by contentrecommendation processing system 300 described above with respect toFIG. 3A, and/or by one or more computing devices at the local office103, such as content server 106. The steps may also be implemented by adistributed computing system, having devices at various locations. Oneor more computing devices may be implemented as a computing device usingthe structure shown in FIG. 2, and may be configured to respond to userrequests for content, such as streaming video or audio. In some aspectsof the present disclosure the content recommendation processing systemmay be operatively connected to and/or integrated within a contentrecommendation system. In such aspects of the present disclosure, thecontent recommendation processing system may provide video on demandservices, and may deliver streaming media, such as sport exhibitions,movies, television shows, radio programs and Internet videos and/oraudio, to a user's consumption or access device (e.g., the displaydevice 112, the gateway interface device 111, the personal computer 114,the wireless device 116, etc.), or any other desired computing device.For brevity, the following description will generally assume that thesteps shown in FIG. 4A are performed by content recommendationprocessing computer 301.

In step 410, the content recommendation processing computer 301, mayreceive information identifying a metadata tag value (t) and a contentitem (p) for which the metadata tag may be applied. For example, themetadata tag value may be a horror metadata tag that has been and/or canbe applied to a movie. As will be described in more detail below, thecontent recommendation processing computer 301 may determine anapplicability score indicating a level of suitability forassigning/annotating the content item with the metadata tag value. Acomputing device, (e.g., the content server 310, the application server107, etc.) may push an identifier for a content item to the contentrecommendation processing computer 301. Another computing device, (e.g.,tag database 306) may push an identifier for a metadata tag value to thecontent recommendation processing computer 301. Alternatively, thecontent recommendation processing computer 301 may receive an identifierfor a metadata tag value and/or a content item from a computing devicein response to a request. For example, a user or system administratormay request the content recommendation processing computer to determinean applicability score for a particular metadata tag and content item byentering a command via an input device.

To better understand the relationship between the applicability scorefor a metadata tag value (t) and a content item (p), two key elementsare involved. The first element, a similarity factor (denoted as“simf(p,t)”), represents the proportion of the similarity between othercontent items that are similar to “p” and that have also been annotatedwith T. The content recommendation processing computer 301 may beconfigured to determine a set of content items that are similar to “p”(denoted as “S(p)”) based on similarity data for “p,” such as similaritydata stored in a similarity matrix. Referring to the example FIG. 4B,the content recommendation processing computer 301 may receive anidentifier for the movie Shrek and an action metadata tag value, asdiscussed in step 410. As shown in FIG. 4B, area 400 depicts the set ofall content items that may be available for consumptions to users of acontent service, such as all of the content items stored at the contentserver 310. As depicted by the area 401 in FIG. 4B, S(p) represents theset of all content items that are similar to Shrek (i.e., content item“p”).

The content recommendation processing computer 301 may determine whethera content item is similar to the movie Shrek by comparing a similarityvalue between Shrek and the content item (i.e., “sim(Shrek, x)”) to apredetermined threshold value. For example, the content recommendationprocessing computer 301 may query a computing device (e.g., programsimilarity module 305) to receive a similarity value between Shrek and acontent item (“x”). The content recommendation processing computer 301may determine a threshold similarity value for comparing the similarityof content items. In some instances, the threshold similarity value maybe provided by a system administrator and/or another computing device.The content recommendation processing computer 301 may determine thatthe content item is similar to Shrek if the similarity value betweenShrek and the content item (i.e., “sim(Shrek, x)”) satisfies thepredetermined threshold value.

As depicted by area 403 in FIG. 4B, T(t) represents the set of allcontent items that have been annotated with the action metadata tagvalue “t.” The content recommendation processing computer 301 may beconfigured to determine a set of content items that have been annotatedwith the action metadata tag value “t” (i.e., “T(t)”). For example, thecontent recommendation processing computer 301 may utilize an identifierfor the action metadata tag value “t” to query a computing device (e.g.,server 310) to receive data indicating one or more other content itemsthat have been annotated with and/or includes the action metadata tagvalue “t.”

As noted above, the similarity factor (denoted as “simf(p,t)”),indicates the proportion of the total level of similarity (asrepresented by a sum of similarity values) between content having athreshold similarity to “p” that have been annotated with “t.” Referringto the example FIG. 4B, the hashed area 402 comprises the subset ofprograms that are similar to the movie Shrek, which have also beenannotated with the action metadata tag. The hashed area 402 similarlycomprises the subset of programs that have been annotated with theaction metadata tag value that are similar to the movie Shrek. In thisexample, the “similarity factor” for Shrek and the action metadata tagvalue comprises the ratio of the sum of similarity values between Shrekand content items within the area 402 to the sum of similarity valuesbetween Shrek and content items within the areas 401 and 402.

In step 420, the content recommendation processing computer 301, maydetermine a similarity factor for the metadata tag value and contentitem determined at step 410. In some embodiments, the similarity factorfor a content item “p” and a metadata tag value “t” is determined usingEquation (1):

simf(p,t)=tagged/total   (1)

where:

the denominator of Equation (1) comprises a “total” parameter that isdetermined using Equation (2):

$\begin{matrix}{{total} = {\sum\limits_{q \in {S{(p)}}}{{similarity}\mspace{14mu} \left( {p,q} \right)}}} & (2)\end{matrix}$

where:

similarity(p,q) comprises a similarity value indicating a level ofsimilarity between the content item “p” and another content item (“q”).

The “total parameter” in Equation (1) indicates the sum of similarityvalues between “p” and each content item (“q”) in the set of othercontent items that are similar to “p.” Referring to the example in FIG.4B, the total parameter would represent the sum of similarity valuesbetween the movie Shrek and each content item in the set of contentitems determined to be similar to Shrek (i.e., the areas 401 and 402).In some embodiments, the tag applicability module 302 may determine theset of content items that are similar to Shrek by querying the programsimilarity module 305. As noted above, the program similarity module 305may retrieve similarity data from a similarity matrix to determine S(p).The program similarity module 305 may be further configured to transmitdata indicating S(p) to the tag applicability module 302, as well ascorresponding similarity values for each content item in S(p).

The numerator of Equation (1) further comprises a “tagged parameter”that is determined using Equation (3):

$\begin{matrix}{{Q = \left\{ q \middle| {q \in {{S(p)}\mspace{14mu} {and}\mspace{14mu} q\mspace{14mu} {is}\mspace{14mu} {tagged}\mspace{14mu} {with}\mspace{14mu} t}} \right\}}{{tagged} = {\sum\limits_{q \in Q}{{similarity}\mspace{14mu} \left( {p,q} \right)}}}} & (3)\end{matrix}$

where “Q” comprises a subset of content items, which includes contentitems that belong to S(p) and content items that have been annotatedwith metadata tag value “t.”

Referring to the example FIG. 4B, the subset of content items comprising“Q” are represented by the content items located within the hashed area402. The “tagged parameter” indicates the sum of similarity valuesbetween content item “p” and each content item in “Q”, i.e., the setcontent items that are both similar to the movie Shrek and have beenannotated with the action metadata tag value. Accordingly, for theexample in FIG. 4B, the tagged parameter comprises the sum of similarityvalues between Shrek and each content item within the hashed area 402.In some embodiments, the tag applicability module 302 may determine theset of content items in the content server 310 that are similar to aparticular content item and that have been tagged with a particularmetadata tag by querying program similarity module 305.

Referring back to Equation (1), the content recommendation processingcomputer 301 may be configured to determine the similarity factor forthe content item and metadata tag value, such as the content item andmetadata tag value identified at step 410 by computing the ratio of thetagged parameter to the total parameter. In the example depicted in FIG.4B, the similarity factor for Shrek and the action metadata tag valuecomprises the ratio of the sum of similarity values between Shrek andcontent items within the area 402 to the sum of similarity valuesbetween Shrek and content items within the areas 401 and 402.

The second element in understanding the relationship between theapplicability score for a metadata tag value (t) and a content item (p)is a tag factor (denoted as tag/(p,t)). The tag factor represents theproportion of content items that have been annotated with the metadatatag value “t” that are similar to the content item “p.” As discussedabove, a similarity between two content items may be determined based onthe similarity value between the two content items satisfying apredetermined threshold. The content recommendation processing computer301 may be configured to determine a number of content items that havebeen annotated with metadata tag value “t” (denoted as “|T(t)|”) byquery a computing device (e.g., the content server 310) to receive dataindicating the number of content items that have been annotated withand/or include the metadata tag value “t.” As depicted by area 403 inFIG. 4B, T(t) represents the set of all content items that have beenannotated with the action metadata tag value “t.” Thus, for thisexample, content recommendation processing computer 301 may determine|T(t)| by determining the number of content items that have beenannotated with the action metadata tag value.

As noted above, the tag factor (denoted as “tagf(p,t)”), indicates aproportion of the content items annotated with “t” that have a thresholdsimilarity “p.” Referring to the example FIG. 4B, the hashed area 402comprises the subset of content items that are similar to the movieShrek, which have also been annotated with the action metadata tag. Thehashed area 402 similarly comprises the subset of content items thathave been annotated with the action metadata tag value that are similarto the movie Shrek. In this example, the tag factor for Shrek and theaction metadata tag comprises the ratio of the number of content itemswithin the area 402 to the number of content items within the areas 402and 403.

In step 430, the content recommendation processing computer 301, maydetermine a tag factor for the metadata tag value and content itemdetermined at step 410. In some embodiments, the “tag factor” isdetermined using Equation (4):

tagf(p,t)=similars/tag total   (4)

where:

the denominator of Equation (4) comprises a “tag total” parameter thatis determined using Equation (5):

tag total=|T(t)|  (5)

where:

|T(t)| comprises the number of content items in T(t), e.g., the numberof content items that have been annotated with a particular metadata tagvalue. In some embodiments, the program similarity module 305 may befurther configured to determine the number of content items in thecontent server 310 that have been annotated with a particular metadatatag value, and may be further configured to transmit to tagapplicability module 302 this data. In still other embodiments, the tagapplicability module 302 may be configured to query a computing device(e.g., the content server 310) to determine the number of content itemsthat have been annotated with a particular metadata tag. As shown inFIG. 4B, |T(t)| represents the number of content items in the areas 402and 403, i.e., the number of content items that have been annotated withthe action metadata tag value.

The numerator of Equation (4) further comprises a “similars” parameterthat is determined using Equation (6):

Q={q|q∈S(p) and q is tagged with t} similars=|Q|  (6)

Similar to Equation (3), the “Q” in Equation (6) comprises a subset ofcontent items, including content items that belong to S(p) and contentitems that have been annotated with metadata tag value “t.” Referring tothe example in FIG. 4B, the subset of content item comprising “Q” isrepresented by the content items located within the hashed area 402. The“similars” parameter indicates the total number of content items thatare both similar to the movie Shrek and have been tagged with the actionmetadata tag value. Referring back to Equation (4), the contentrecommendation processing computer 301 may be configured to determinethe tag factor for the content item and metadata tag value, such as thecontent item and metadata tag value identified at step 410 by computingthe ratio of the number of content items within the area 402 to thenumber of content items within the areas 402 and 403.

In step 440, the content recommendation processing computer 301, maydetermine an applicability score for the metadata tag value and contentitem determined at step 410. The content recommendation processingcomputer 301 may be configured to determine an applicability score(denoted as “weight(p,t)”) for a metadata tag value and a content itembased on the similarity factor, as determined at step 420, and the tagfactor as determined at step 430.

In some embodiments, an applicability score is determined using Equation(7):

weight(p,t)=simf(p,t)·tagf(p,t)   (7)

where the applicability score for a metadata tag and content itempairing (denoted as “weight(p,t)”) comprises the product of thesimilarity factor and tag factor for the metadata tag and content itempairing. In such embodiments, the tag applicability module 302 may beconfigured to generate the applicability score for a metadata tag value“t” and a content item “p” by utilizing Equations 1-7 described above.The tag applicability module 302 may be configured to store thegenerated applicability score in the weighting database 303. In someaspects of the present disclosure, the tag applicability module 302 maygenerate applicability scores for each metadata tag stored in the tagdatabase 306 with respect to one or more content items or group ofcontent items stored in the content server 310. In some aspects of thepresent disclosure, the weighting database 303 may store applicabilityscores for every potential metadata tag and content item pairing in viewof the plurality of metadata tag values stored in the tag database 306and the plurality of content items stored in the content server 310.

Referring back to FIG. 3A, the content recommendation module 304 may beconfigured to generate content recommendations for a user or group ofusers. In some aspects of the present disclosure, the contentrecommendation processing computer 301 may receive input from acomputing device (e.g., the client device 320) indicating a request fora particular type of content item. For example, a user may utilize anelectronic program guide (or other suitable user interface) via theclient device 320 to request the display of a listing of content itemshaving a specified metadata field and/or metadata tag value, such ashorror movies. In this example, the content recommendation processingcomputer 301 may query the content recommendation module 304 to generatea plurality of content items corresponding to the user content request.Referring back to the example above, if a user requests a list ofrecommended horror movies, the content recommendation module 304 mayquery weighting database to determine one or more content items thathave been annotated with a horror metadata tag value and that have acorresponding applicability score that exceeds a predetermined thresholdvalue.

Additionally or alternatively, the content recommendation module 304 maybe configured to modify the display of recommended content items and/orprogramming content that are provided to a user. For example, contentrecommendation module 304 may sort/rank content items annotated with ahorror metadata tag value by ascending order of correspondingapplicability scores. As a result, content items having the largestapplicability scores for their respective horror metadata tag values,and thus are most similar to the prototypical horror movie, may bepresented at the top of a list of preferred content items for the user.By contrast, content items having the lowest applicability scores fortheir respective horror metadata tag values, and thus are leastrepresentative of the prototypical horror movie, may be placed towardthe end of such list, or not presented to the user at all. For instance,the content recommendation module 304 may not present to the user anycontent items annotated with a horror metadata tag value having anapplicability score below a predetermined threshold value. In suchexamples, in response to the content request for horror movies, contentrecommendation module 304 may return a predetermined number of contentitems having the highest relative applicability scores associated withtheir respective horror metadata tag value.

In some aspects of the present disclosure, the content recommendationmodule 304 may be configured to determine a plurality content items thathave similar metadata tag values and applicability scores as a targetcontent item. For example a user at the client device 320 may indicate aviewing preference for a particular content item (e.g., target contentitem), and may request the content recommendation computer 301 todetermine one or more content items that are similar to (or “more like”)the target item. In this example, content recommendation module 304 mayidentify and store in memory (or other suitable storage) each metadatafield and corresponding metadata tag values for the target content item.The content recommendation module 304 may query the weighting database303 to identify a content item annotated with similar metadata tagvalues and corresponding applicability scores as the target contentitem. For example, the content recommendation module 304 may query theweighting database 303 to identify a content item having a firstmetadata tag value that has been assigned an applicability score that iswithin a threshold value of the applicability score assigned to the samemetadata tag value for the target content item.

In other aspects of the present disclosure, content recommendationmodule 304 may utilize content consumption history and/or user contentpreferences/rankings of a user to determine recommended content items.The content recommendation module 304 may generate contentrecommendations based on a consumption history of the user requestingcontent recommendations, the consumption history of one or more otherusers or groups of users, and/or other user content consumptionpreferences. For example, the content recommendation module 304 may beconfigured to analyze the metadata field and corresponding metadata tagvalues of a content item (e.g., target content item) that has beenconsumed (or reserved) by a user more than a threshold number ofinstances within a predetermined time period (e.g., 1 week, 1 month,etc.). The content recommendation module 304 may be further configuredto determine one or more content items annotated with similar metadatatag values and corresponding applicability scores as the target contentitem. As another example, the content recommendation module 304 may beconfigured to analyze the metadata field and corresponding metadata tagvalues of a content item for which the user has indicated a viewingpreference. A user may indicate a viewing preference for a content itemin various ways without departing from the scope of the presentdisclosure, such as by providing a high or low ranking to the contentitem, marking the content item with either a “favorite” or “dislike”indicator, and the like. The content recommendation system 304 mayrecommend content items to the user that have similar metadata tagvalues and corresponding applicability scores as the target contentitem, and may be further configured to deprioritize and/or suppresscontent recommendations for content items that are associated withcontent items for which the user has indicated a negative viewingpreference, such as content items having low rankings and/or areassociated with a dislike indicator.

As discussed above, requiring a system administrator to determine andassess the suitability of annotating a content item with a metadata tagvalue can be time consuming and error prone. These issues may becompounded when attempting to annotate each metadata field of a contentitem with a plurality of available metadata tag values, or whendetermining applicability scores for a large number of content items.For example, a content service provider may desire to add one or morenew content items, such as a newly released movie, to a plurality ofcontent items available to users for consumption. Each new content itemmade available to users may contain a limited amount metadata (if any),or may only contain metadata for a limited set of metadata fields.Accordingly, when adding a new content item to a content server and/orwhen supplementing/refreshing metadata for content items in the contentserver, the content provider may require a recommendation system capableof assigning metadata tag values to a plurality of metadata fields of acontent item. Additionally or alternatively, the content provider mayrequire the recommendation system to update applicability scores forexisting content items based on the new content item and associatedmetadata.

As discussed above with respect to FIG. 4B, the content recommendationprocessing system may be configured to generate an applicability scorefor a metadata tag value and a content item indicating a level/degree ofsuitability for annotating the content item with the metadata tag value.In additional aspects of the present disclosure, the contentrecommendation processing system may be configured to assign metadatatag values to a metadata field of a content item based on applicabilityscores determined for the respective metadata tag values.

FIG. 5 is an exemplary flowchart of a process for generating andassigning metadata tag values and corresponding applicability scores tocontent items according to one or more illustrative aspects of thedisclosure. The generation of metadata tag values for various metadatafiles of a content item may be implemented on its own by a contentrecommendation processing system to add and/or supplement metadata for aplurality of content items. The content recommendation processing systemmay generate applicability scores for each metadata tag value andcontent item to determine the suitability of annotating the content itemwith the metadata tag value. As will be discussed in further detailbelow, the generated weight scores may be further utilized by thecontent recommendation system to suggest content to a user, and/or maybe combined with other recommendation systems and entertainment programguides.

The steps shown in FIG. 5 may be performed by a content recommendationprocessing system (e.g., by content recommendation processing system 300described above with respect to FIG. 3A). For example, the steps may beimplemented by the content recommendation processing system computer301. The steps may also be implemented by a distributed computingsystem, having devices at various locations. For brevity, the followingdescription will generally assume that the steps shown in FIG. 5 areperformed by the content recommendation processing computer 301. In step501, the content recommendation processing computer 301 may performvarious system configuration steps. This configuration process mayinclude registering user consumption or access devices with a contentprovider. For example, this registration step may include receivinginformation (e.g., a username, password, device ID, etc.) for setting upa user account with a content provider. When setting up the account, auser may specify content, and/or device preferences that may determine,at least partially, which content items are subsequently provided to theuser. Additionally or alternatively, users may have the option togenerate and/or modify existing metadata fields and tag information fora plurality of content items. For example, a user may create a newmetadata tag value to describe a characteristic of a content item. Asanother example, the user may choose one or more metadata tag valuesfrom a database to associate with (and/or annotate) a particularcontent. Additionally or alternatively, a user may establish (orindicate) their content preferences by providing rankings for certaincontent items and/or metadata fields of a content item. For instance,within the genre metadata field of a content item, a user may have theoption to provide rankings for a plurality of metadata tag values (e.g.,horror, action, adventure, comedy, etc.).

Various metadata tag values (and/or other information) may be includedwithin (and/or annotated to) a metadata field of a content items withoutdeparting from the scope of the present disclosure. One or more clientor access devices associated with a user may be configured to permit theuser to assign rankings to various metadata files and corresponding tagvalues, as described above. The one or more client or access devicesutilized by the user may also be configured to transmit user rankings toa content provider. In some aspects of the present disclosure, the oneor more client or access devices associated with the user may also beconfigured to generate metadata tag values, and annotate a content itemwith the metadata tag values.

As discussed above a plurality of metadata tag values may be stored in atag database and used to describe various characteristics of a contentitem or a portion of a content item. In some aspects of the presentdisclosure, the metadata tag values stored in the database may comprisetag values created by system administrators to describe variouscharacteristics of the plurality of content items made available tousers for consumption. In step 503, the content recommendationprocessing computer 301 may determine and/or identify one or moreavailable metadata tag values that are associated with available contentitems, such as content items stored in a content server. In someinstances, the content recommendation processing system mayidentify/process a universe of available metadata tag values that may beused to characterize the subject matter of content items.

In other aspects of the present disclosure, the content recommendationprocessing computer 301 may determine/identify a plurality of metadatatag values that have been annotated to content items made available tousers for consumption. In such instances, the plurality of existingmetadata tag values associated with content items on the content servermay comprises a subset of the universe of available metadata tag valuesthat may be used to characterize a content item. As described above, auser (or a system administrator) may have the option of creating a newmetadata tag values for a particular metadata field of a content item.For example, the user may be prompted via a user interface (e.g.,electronic program guide) to create a new metadata tag value for acontent item by providing input characterizing the subject matter and/ora metadata field of the content item. The content recommendationprocessing computer 301 may receive textual data characterizing thesubject matter of the content item, and may generate a metadata tagvalue based on the textual data. In this example, a user maycharacterize the genre of a content item as “Paranormal Romance,” andthe content recommendation processing computer 301 may generate a newmetadata tag value, named “Paranormal Romance,” that may be used todescribe the genre of content items. At step, 503, the contentrecommendation processing computer 301 may store the newly createdmetadata tag value in memory, a database, or other suitable storagemeans for subsequent use by the content recommendation processingcomputer 301.

In some aspects of the present disclosure, a content service providermay not desire to assign metadata tag values to content items when thetag values have rarely been used to describe other content items.Metadata tag values that have been used to characterize a limited numberof content items may be associated with obscure and/or narrow subjectmatter, and as such, may not provide the best indication of the subjectmatter of a content item. For example, a metadata tag value of “menwalking past each other while riding flamingos” within a genre metadatafield of a content item may accurately describe the genre and subjectmatter of a limited number of content items, but would likely fail todescribe a large set of content items. The content recommendationprocessing computer 301 may receive input, from an input device,indicating parameters for a metadata tag value. These parameters may beconfigured to restrict which metadata tag values are considered and/orretrieved by the content recommendation processing computer 301. Forexample, the content service provider may desire that the contentrecommendation processing system exclude metadata tag values that havenot been annotated or assigned to a minimum threshold number of contentitems. For example, the content recommendation processing computer 301may receive a parameter indicating that the threshold minimum number ofcontent items is a value within the range of four (4) to sixteen (16)content items. Any threshold number of minimum metadata tagannotations/assignments may be utilized by the content recommendationprocessing computer 301 when determining which metadata tag values toconsider/retrieve from a database. In some instances, a computing devicemay push parameters for particular metadata tag values to contentrecommendation processing computer 301, such as a parameter indicatingthe minimum threshold number of content items. Alternatively, thecontent recommendation processing computer 301 may receive a parameterfor a metadata tag value from a computing device in response to arequest.

As discussed above, a content service may offer a user a plurality ofcontent items for consumption. To more efficiently organize the contentitems in a manner that is easily comprehended by the user, content itemsmay be characterized by content characteristics embodied by theirrespective metadata fields and tag values. However, each content itemavailable to a user may include only a fraction of the possible metadatatag values utilized by the content recommendation processing system tocharacterize the subject matter and/or aspects of a content item.Additionally, applicability scores may be assigned to an even smallersubset of metadata tag values (if any) for new and/or available contentitems. This is particularly relevant within the context ofsparsely-viewed or new content items, which may have fewer metadata(and/or assigned applicability scores), as compared to content itemsthat have been made available for consumption to users over an extendedperiod of time and/or have been consumed more frequently by a largernumber of users. Accordingly, the content recommendation processingsystem may need to identify content items that may require additionalmetadata.

In step 505, the content recommendation processing computer 301 mayreceive input identifying content items to annotate with metadata and/orapplicability scores. The content recommendation processing computer 301may receive input from a computing device, such as the client device320, identifying one or more content items (or groups of content items)for which the content recommendation processing computer 301 maygenerate metadata tag values and/or assign corresponding applicabilityscores. For example, a content service provider planning to introducenew content items (e.g., newly-released movies, a new season oftelevision show, etc.) may need to generate metadata tag values and/orapplicability scores for new content items. As another example, acontent service provider may identify a subset of content itemscurrently available to users that do not have applicability scores, oravailable content items that have limited metadata. Accordingly, asystem administrator may utilize an input device to provide input to thecontent recommendation processing computer 301 identifying those contentitems (e.g., new content items, content items having a low number ofmetadata tag values) that may require metadata tag annotations and/orapplicability score assignments.

In some aspects of the present disclosure, a content service providermay not desire to provide metadata tag values and/or applicability scoreassignments to content items that have rarely been consumed by usersand/or not consumed by a threshold number of users. Content items thathave been rarely consumed may represent random/statistical outliers thatdo not provide the best indication of how similar that content item isto other content items. The content recommendation processing computer301 may receive input indicating content item viewing parameters, whichmay be configured to restrict which content items are considered by thecontent recommendation processing system for metadata tag annotationsand/or applicability score assignments. In some embodiments, the contentservice provider may desire the content recommendation processing systemto exclude those content items that have not been consumed by more thaneight (8) users. Any minimum threshold number of users that haveviewed/consumed a content item may be utilized by the contentrecommendation processing system when determining which content itemsmay require a metadata tag annotations and/or an applicability scoreassignment.

In still other aspects of the present disclosure, the contentrecommendation processing computer 301 may receive input identifyingcontent items associated with a user content query. The contentrecommendation processing computer 301 may receive input from acomputing device, such as the client device 320, identifying one or morecontent items (or groups of content items) related to a user contentsearch. For example, a user may request via an electronic program guide(or other suitable interface) a listing of action movies available forconsumption. The content recommendation processing computer 301 mayreceive the content query from the client device 320, and may query acontent server for a plurality of content items that include a metadatafield having a particular metadata tag value, such as an action metadatatag value. As will be discussed in more detail below, the contentrecommendation processing computer 301 may analyze content itemsresponsive to a user content query (e.g., content items that include theaction metadata tag value), and may generate updated metadata tagannotations and/or updated applicability score assignments for contentitems in accordance with the query. In other instances, the contentrecommendation processing computer 301 may generate updated metadata tagvalues and/or updated applicability score assignments for content itemsresponsive to other user content queries and/or in response to otheruser events, such as consuming a threshold number of content items,indicating a content item preference, recording programming content,identifying new content items available for consumption, and the like.

In step 507, the content recommendation processing computer 301 maygenerate and/or retrieve similarity data for a plurality of contentitems. For example, the content recommendation processing computer 301may generate and/or retrieve similarity data for content itemsidentified at step 505. As discussed above, the content recommendationprocessing system may be configured to analyze user content consumptionhistory data to determine similarities between content items consumed bya plurality of users. The content recommendation processing computer 301may generate and/or retrieve similarity data (e.g., similarity values)for a plurality of content items from a similarity matrix, such as thesimilarity matrix described above with respect to FIG. 3B. In someembodiments, the content recommendation processing computer 301 mayrequest from a computing device similarity data for a plurality ofcontent times, such as the content items identified at step 505. In someinstances, the computing device may push similarity data to the contentrecommendation processing computer 301. In other instances, the contentrecommendation processing computer 301 may receive similarity data froma computing device in response to a request. For example, the contentrecommendation processing computer 301 may provide the computing device,e.g., program similarity module 305, with identifiers for a firstcontent item and a second content item, and the computing device maytransmit to the content recommendation processing computer 301 asimilarity value between the first and second content items.

As will be discussed in more detail below, in some aspects of thepresent disclosure, the content recommendation processing computer 301may utilize the similarity data and existing metadata information forcontent items (e.g., the content items identified at step 505), suchthat for any given content item and metadata tag value, the contentrecommendation processing computer 301 may compute an applicabilityscore indicating a degree of suitability for annotating the content itemwith the metadata tag value. As discussed above, the applicability scorefor a content item and metadata tag value may represent a level ofsimilarity between the content item and a subset of content that alsohave been assigned the particular metadata tag value.

As previously noted, a determined similarity value (and/or a similarityvalue retrieved from a similarity matrix) may fall within a wide rangeof values, and the content recommendation processing computer 301 may beconfigured to adjust/normalize determined similarity values and/orsimilarity values stored in or retrieved from a similarity matrix. Forexample, rather than determining “ratio(p,q)” (i.e., the ratio of thefraction/proportion of total users that have consumed a first contentitem (p) and a second content item (q) to the fraction/proportion ofusers that have consumed the second content item), the contentrecommendation processing computer 301 may determine a similarity valuebetween the two content items using a probability difference, which maybe expressed as:

delta(p,q)=P(q|p)−P(q)

or:

delta(p,q)=P(q)·(ratio(p,q)−1)

In other aspects of the present disclosure, the content recommendationprocessing computer 301 may adjust and/or modify a similarity valuebetween two content items based on other information, such assimilarities in the textual descriptions of the content items. Forexample, in some embodiments, the content recommendation processingsystem (and/or another suitable computing device) may analyze closedcaption information, abstract information, program descriptioninformation, and other textual information associated with a contentitem to determine whether two programs are similar. In one of theseembodiments, the content recommendation processing system may utilize anatural language processor (and/or any other suitable linguisticprocessor) to analyze the textual information associated with a contentitem. The content recommendation processing system may be configured todetermine similarities between content items using the results of thetextual analysis.

In still other aspects of the present disclosure, the contentrecommendation processing computer 301 may adjust and/or modify asimilarity value between two content items based on similarities invideo/visual data of two content items. In some embodiments, the contentrecommendation processing computer 301 (and/or another suitablecomputing device) may analyze video/visual data associated with acontent item to determine similarity between two content items. In oneof these embodiments, the content recommendation processing system mayutilize a specially-programmed video processor, such as a processor incontent recommendation processing 301 (and/or any other processorcapable of analyzing video data for relevant content), to analyze thevideo/visual data of a content item. The content recommendationprocessing system may utilize the specially-programmed video processorto analyze video data for content items with reference to profileinformation to select segments of interest within the content item thatare associated with predetermined events of significance signifying thatthe content item is associated with a particular metadata tag value. Forexample, identifying predetermined events (e.g., events corresponding toexplosions, gun shots, car chases and the like), within video data for afirst content item may infer that the first content item is associatedwith an action metadata tag value. Accordingly, the contentrecommendation processing computer 301 may modify/adjust upward thesimilarity value between the first content item and another content itemannotated with an action metadata tag value and/or another content itemhaving a threshold applicability score for the action metadata tagvalue. Various types of identifying events may be utilized by thecontent recommendation processing system to identify correspondingmetadata tag values and/or metadata fields for a content item based onvideo data associated with the content item without departing from thescope of the present disclosure.

At step 509, the content recommendation processing computer 301 maybegin a loop that is performed for one, some, or all of the contentitems identified at step 505. For example, a system administrator maydesire to conduct a large-scale expansion of metadata tag values for aplurality of content items stored on a server in order toreplace/supplement metadata, confirm existing metadata annotations,and/or to generate metadata tag annotations for the plurality ofcontent. As will be explained in more detail below, as a result of thelooping in steps 509, 512, and 513, the content recommendationprocessing computer 301 may generate, for each content item (“P”)analyzed at step 509, a separate applicability score (denoted“weight(p,t)”) may be generated for each metadata tag value (“T”)available to describe/characterize a metadata field of content item P.The content recommendation processing computer 301 may utilize anapplicability score determined for the metadata tag value and contentitem to assess the suitability of annotating the content item with theparticular metadata tag value.

At step 511, the content recommendation processing computer 301 maydetermine a plurality of content items that are similar to the contentitem being analyzed at step 509. The content recommendation processingcomputer 301 may utilize similarity data retrieved (and/or generated) atstep 507 to determine whether one or more content items are similar tothe content item being analyzed at step 509. The content recommendationprocessing computer 301 may determine the plurality of content itemsthat are similar to the content item being analyzed at step 509 in amanner similar to step 420. As discussed above, the contentrecommendation processing computer 301 may determine that a firstcontent item is similar to the content item being analyzed at step 509when a similarity value between the first content item and the contentitem being analyzed at step 509 satisfies a predetermined thresholdvalue.

At step 512, the content recommendation processing computer 301 maybegin a loop that is performed for a plurality of metadata fields. Insome embodiments, the computer 301 may begin a loop that is performedfor one, some, or all metadata fields retrieved at step 503. Forexample, a system administrator may desire to update metadata tag valuesfor one, some, or all metadata fields of the content items received atstep 505. The content recommendation processing computer 301 may receiveinput from a system administrator identifying the metadata fields towhich generated metadata tag values should be assigned/annotated. Inother instances, the content recommendation processing computer 301 maygenerate metadata tag values for all available metadata fields of acontent item.

At step 513, the content recommendation processing computer 301 maybegin a loop that is performed for a plurality of metadata tag values.In some embodiments, the computer 301 may begin a loop that is performedfor one, some, or all of the metadata tag values determined (and/orretrieved) at step 503. The content recommendation processing system maydetermine a universe of metadata tag values available for annotatingcontent items. In some instances the content recommendation processingsystem may determine a plurality of metadata tag values available tocharacterize the metadata field being analyzed at step 512. In someaspects of the present disclosure, the content recommendation processingsystem may be configured to exclude from consideration those metadatatag values that have not been assigned to (and/or associated with) athreshold number of content items. For example, the contentrecommendation processing system may exclude from consideration metadatatag values that have not been annotated to (and/or associated with) atleast 4 content items. Any number of minimum content items may beutilized by the content recommendation processing system whendetermining which metadata tag values to exclude from considerationwithout departing from the scope of the present disclosure.

At step 517, the content recommendation processing computer 301 maydetermine/generate an applicability score for the metadata tag valuebeing analyzed at step 513. The content recommendation processingcomputer 301 may determine an applicability score for the metadata tagvalue and the content item being analyzed at step 509 in a mannersimilar to step 440. To determine the applicability score, the contentrecommendation processing computer 301 may be configured to determine asimilarity factor and a tag factor for the metadata tag value and thecontent item in a manner similar to steps 420 and 430, respectively.

At step 518, the content recommendation processing computer 301 maydetermine whether the applicability score determined at step 517satisfies a threshold value. As noted above, given that theapplicability score for a particular metadata tag value and content itemindicates the degree of similarity between the content item and othercontent items that have also been annotated with the particular metadatatag, the content recommendation processing computer 301 may determinethe appropriateness or applicability of annotating the content item withthe particular metadata tag value based on the determined applicabilityscore. The content recommendation processing computer 301 may determinewhether to apply and/or annotate the metadata tag value (being analyzedat step 513) to the content item (being analyzed at step 509) bycomparing the applicability score determined for the metadata tag valueand the content item pairing to a threshold value.

For example, when determining whether to assign a comedy metadata tagvalue to the genre metadata field of a movie, the content recommendationprocessing computer 301 may compare the applicability score for thecomedy metadata tag value and movie to a minimum threshold applicabilityscore. In response to a request from the content recommendationprocessing computer 301, another computing device may push dataindicating a minimum threshold applicability score for determiningwhether to annotate a content item with a metadata tag. In other aspectsof the present disclosure, the content recommendation processingcomputer 301 may receive input (e.g., input from a system administrator)identifying a minimum threshold applicability score. Various thresholdvalues may be utilized by the content recommendation processing computer301 when determining whether to annotate a content item with a metadatatag value without departing from the scope of the present disclosure.For example, the content recommendation processing computer 301 mayutilize a first threshold for assessing whether to annotate contentitems with a first metadata tag value (e.g., horror) and may utilize adifferent threshold for assessing whether to annotate the content itemsa second metadata tag value (e.g., comedy).

Additionally or alternatively, the content recommendation processingcomputer 301 may determine whether the determined applicability scoresatisfies a preliminary threshold applicability score. The preliminarythreshold applicability score may be used to determine whether anapplicability score is statistically significant for consideration bythe content recommendation processing system. In some instances thecontent recommendation processing computer 301 may be configured toexclude from consideration those applicability scores that do not exceeda preliminary threshold value. For example, the content recommendationprocessing computer 301 may exclude from consideration applicabilityscores that fall below 0.001. In this example, the contentrecommendation processing computer 301 may determine that applicabilityscores below 0.001 represent a statistically random outlier and shouldnot be assigned to and/or associated with a particular metadata tagvalue and content item. Although not depicted in FIG. 5, if the contentrecommendation processing computer 301 determines that the applicabilityscore determined at step 517 fails to satisfy the preliminary thresholdvalue, the method may proceed to step 513. If the content recommendationprocessing computer 301 determines that the applicability scoredetermined at step 517 satisfies the preliminary threshold value, thecontent recommendation processing computer 301 may compare theapplicability score determined at step 517 to a second threshold asdescribed above (e.g., comparing the applicability score to a thresholdvalue to determine whether to apply and/or annotate the metadata tagvalue to a content item). Various minimum threshold applicability scoresmay be utilized by the content recommendation processing system whendetermining which generated applicability scores should be assigned to ametadata tag value and content item without departing from the scope ofthe present disclosure.

At step 518, if the applicability score determined at step 517 satisfiesthe threshold, then the method may proceed to step 519 where the contentrecommendation processing computer 301 may annotate/apply the metadatatag value being analyzed at step 513 to the metadata field (beinganalyzed at step 512) of the content item being analyzed at step 509.The content recommendation processing computer 301 may annotate and/orassociate the metadata tag value with a metadata field of a content itemin various manners without departing from the scope of the presentdisclosure. For example, the content recommendation processing computer301 may store data indicating the metadata tag value in data field(e.g., metadata field) of a content item. Additionally or alternatively,the content recommendation processing computer 301 may store themetadata tag value in a database, and may be configured to associatewithin the database the metadata tag value and the metadata field of thecontent item. The content recommendation processing computer 301 maystore the applicability score determined at step 517 in memory or anyother suitable storage means. For example, the content recommendationprocessing computer 301 may store the applicability score in a database(e.g., database 303). In some embodiments, the content recommendationprocessing computer 301 may transmit to a computing device theapplicability score determined at step 517.

As discussed above with respect to steps 518 and 519, in some aspects ofthe present disclosure, the content recommendation processing computer301 may propose or recommend a metadata tag value to be assigned to ametadata field of a content item when an applicability score generatedfor the metadata tag score exceeds a threshold value. Additionally oralternatively, the content recommendation processing computer 301 maygenerate an alert to assign a metadata tag value to a metadata field ofa content item when an applicability score generated for the metadatatag value exceeds a threshold value. For example, the contentrecommendation processing computer 301 may generate an alert (and/orother suitable notification, communication, etc.) indicating that agenerated applicability score for a metadata tag value exceeds thepredetermined threshold value. The content recommendation processingcomputer 301 may prompt a system administrator to assign and/orassociate the metadata tag value with the content item. Additionally oralternatively, the content recommendation processing computer 301 mayassign and/or associate the metadata tag value with the content item.

In other aspects of the present disclosure, the content recommendationprocessing computer 301 may propose or recommend one or more metadatatag values to be assigned to a content item utilizing a sorting/rankingof applicability scores. In some embodiments, the content recommendationprocessing computer 301 may be configured to sort/rank metadata tagvalues by applicability scores for a particular metadata field of acontent item. For example, the content recommendation processingcomputer 301 may rank applicability scores for metadata tag values(e.g., horror, action, comedy, drama, etc.) within the genre metadatafield of a content item. The content recommendation processing computer301 may be configured to identify one or more metadata tag values havingthe highest generated applicability scores for the particular metadatafield of the content item, and may be further configured to generate anotification or communication recommending that one or more of theidentified metadata tag values be assigned to and/or associated with thecontent item.

For example, as shown below, Table 1 depicts several metadata tag valuesand corresponding applicability scores generated by the contentrecommendation processing computer 301 for four different metadatafields of the movie Shrek (i.e., genre, tone, theme, and keyword). Ascontext, the movie Shrek is a computer-animated film where an ogre'speaceful world is upended by the invasion of magical creatures who werebanished from their lands by a villainous lord. To save their land, andhis, the ogre agrees to rescue a princess from the villainous lord andtake back the kingdom.

As further shown in Table 1, the metadata tag values in bold representexisting metadata tag values that were already assigned to and/orassociated with the movie Shrek, for example, metadata tag values thatwere manually assigned to the content item by system administrators.

TABLE 1 Genre Theme Tone Keyword Animated 0.2693 Imaginary Kingdom0.0708 Fanciful 0.0872 Friendship 0.0696 Family Ent. 0.2519 DaringRescue 0.0614 Humorous 0.0851 Hero 0.0498 Children Ent. 0.2045 FantasyLand 0.2693 Rousing 0.0475 Danger 0.0461 Fantasy 0.1142 Heroic Mission0.2693 Stylized 0.0279 Partner 0.0266 Action/Adventure 0.0652 Curses &Spells 0.2693 Witty 0.0081 Fantasy-World 0.0231 Comedy 0.0596 Fall inLove 0.2693 Quirky 0.00667 Pop Culture 0.0090 Adaption 0.0105 OppositesAttract 0.2693 Sentimental 0.0047 Dragon 0.0010

As shown above, the content recommendation processing computer 301 maybe configured to rank/sort by applicability score the metadata tags fora metadata field of the movie Shrek. As shown above, several metadatatag values available to characterize/describe the movie Shrek were notpreviously assigned to the content item by system administrators. Thecontent recommendation processing computer 301 may be configured torecommend one or more metadata tag values be assigned to and/orassociated with the “genre” metadata field of the movie Shrek that havenot been previously assigned to the content item.

In some embodiments, the content recommendation processing computer 301may be configured to assign metadata tag values to the genre metadatafield of the movie Shrek that exceed a predetermined thresholdapplicability score, in a similar manner as step 518. For example,referring to Table 1 above, the content recommendation processingcomputer 301 may not assign metadata tag values (within the genremetadata field) to the movie Shrek if the generated applicability scorefor the metadata tag fails to exceed 0.1000. In this example, thecontent recommendation processing computer 301 may recommend assigningthe “Family Entertainment” and “Fantasy” metadata tag values to thegenre metadata field of the movie Shrek, given that the “ChildrenEntertainment” and “Fantasy” metadata tag values have already beenassigned to the movie. Various threshold applicability scores may beutilized by the content recommendation processing computer 301 whendetermining which metadata tag values to recommend assigning to acontent item without departing from the scope of the present disclosure.In some embodiments, the content recommendation processing computer 301may receive user input indicating a threshold applicability score forone or more content items and/or corresponding metadata field. In one ofthese embodiments, a user may interact with an interface element (e.g.,a slider, menu, etc.) of an electronic program guide and/or othersuitable user interface to indicate the threshold applicability scorefor recommending that a metadata tag value be assigned to a contentitem. As will be described in more detail below with respect to FIG. 7,the content recommendation processing computer 301 may utilize rankingsof metadata tag applicability scores to determine contentrecommendations for users and/or to affect the order of content itemspresented to a user in an electronic program guide.

In some aspects of the present disclosure, the content recommendationprocessing computer 301 may be configured to recommend assigning to acontent item a threshold number of the highest ranked metadata tagvalues for a particular metadata field based on generated applicabilityscores. For example, referring back to Table 1 above, the contentrecommendation processing computer 301 may recommend assigning the movieShrek the four (4) highest-ranked metadata tag values for each metadatafield. In this example, the content recommendation processing computer301 may generate a notification or communication recommending that the“Imaginary Kingdom” and “Daring Rescue” metadata tag values be assignedto the “theme” metadata field of the movie Shrek, given that the“Fantasy Land” and “Heroic Mission” metadata tag values have alreadybeen assigned to the “theme” metadata field for the movie Shrek.

If the applicability score determined at step 517 satisfies thethreshold, then the method may proceed to step 513 to continue the loopuntil each metadata tag value has been processed by the contentrecommendation processing computer 301. The content recommendationprocessing computer 301 may generate and/or assign applicability scoresfor all available metadata tag values associated with the metadata fieldbeing analyzed at step 512. In some aspects of the present disclosure,at step 513, the method may continue the loop until one, some, or all ofthe metadata tag values determined (and/or retrieved) at step 503 havebeen processed. When all of the metadata tag values have been processed,the method may return to step 512 to continue the loop until each of theplurality of metadata fields have been processed by the contentrecommendation processing computer 301. At loop 512, the contentrecommendation processing computer 301 may generate applicability scoresand/or assign metadata tag values to all available metadata fields forthe content item being analyzed at step 509. In some aspects of thepresent disclosure, at step 512, the method may continue the loop untilone, some, or all of the metadata fields determined (and/or retrieved)at step 503 have been processed. When all of the metadata fields havebeen processed, the method may return to step 509 to continue the loopuntil each of the plurality of content items have been processed.

As discussed above, one or more content items made available to usersfor consumption may already include existing metadata. For example, auser or system administrator may have manually annotated one or morecontent items (e.g., newly-released content items) with metadata, suchas metadata fields and corresponding metadata tag values. Rather thangenerate new metadata for a content item, a service provider or systemadministrator may desire to confirm/assess the applicability of existingmetadata for content items. For example, the service provider may wantto asses/confirm whether a horror metadata tag value currently assignedto a content item is appropriate.

Furthermore, as a content item is consumed by a larger number of users,the service provider may receive feedback from users regarding theappropriateness/applicability of a metadata tag value assigned to acontent item. The service provider may desire to update or modifymetadata tag values for content items based on such user feedback. Asdiscussed above, the similarity between a first content item and asecond content item may be based in part on the number of users thatconsumed a first item and a second item and the fraction of users thatconsumed the second item. Thus, as additional users become members of acontent service and as the content consumption/preferences for new andexisting users change over time, the applicability score for a metadatatag value and content item may also vary over time.

FIG. 6 shows an exemplary flowchart of a process for calibratingapplicability scores and updating metadata tag values for content itemsaccording to one or more illustrative aspects of the disclosure. As willbe discussed in further detail below, the content recommendationcomputer 301 may be configured to update and/or calibrate applicabilityscores based on feedback data, and may be further configured to utilizeapplicability scores to determine whether existing metadata informationfor a content item is still applicable and/or appropriate. The stepsshown in FIG. 6 may be performed by a content recommendation processingsystem (e.g., by content recommendation processing system 300 describedabove with respect to FIG. 3A). For example, the steps may beimplemented by the content recommendation processing system computer301. For brevity, the following description will generally assume thatthe steps shown in FIG. 6 are performed by content recommendationprocessing computer 301.

In step 601, the content recommendation processing computer 301, mayreceive information identifying a metadata tag value and a content item.As will be described in more detail below, the content recommendationprocessing computer may determine and/or retrieve an applicability scorebased on the metadata tag value and content item. In some instances, acomputing device may push an identifier for a content item and/or ametadata tag value to the content recommendation processing computer301. Alternatively, content recommendation processing computer 301 mayreceive an identifier for a metadata tag value and/or a content itemfrom a computing device in response to a request. For example, a user orsystem administrator may request the content recommendation processingcomputer to determine an applicability score for a particular metadatatag and content item by entering a command via an input device. Thecontent recommendation processing computer 301 may receive dataidentifying a plurality of metadata tags and corresponding contentitems.

In step 603, the content recommendation processing computer 301 maydetermine and/or retrieve an applicability score for the metadata tagvalue(s) and content item(s) received/identified at step 601. Thecontent recommendation processing computer 301 may determine anapplicability score for the metadata tag value and content item in asimilar manner as in step 517. In some instances, a computing device maypush an applicability score for the metadata tag value and content itemto the content recommendation processing computer 301. In otherinstances the content recommendation processing computer 301 may receivefrom a computing device an applicability score for the metadata tagvalue and content item in response to a request.

At step 605, the content recommendation processing computer 301 maycalibrate applicability score(s) for the metadata tag value(s) andcontent item(s) determined at step 601. As discussed above, over time,the content recommendation processing computer 301 may receive and/orcollected feedback data from users indicating content preferences, andthe content recommendation processing computer 301 may be configured toadjust applicability scores for metadata tag values based on thecollected information. In some aspects of the present disclosure, thecontent recommendation processing system may receive input indicating anaccuracy (or correctness) of metadata tag values that have been assignedto content items, such as tag values that have been manually assigned toa content item and/or tag values that have been assigned to contentitems in a similar manner as steps 509-519. For example, a contentprovider may retrieve user feedback via a user interface (e.g.,electronic program guide) indicating an accuracy of metadata tag valuesassigned to one or more metadata fields of a content item. In thisexample, the user may be prompted to identify whether a first metadatatag value accurately reflects the subject matter of an available (and/orconsumed) content item. In another example, the content recommendationprocessing computer 301 may generate and display to a user, via the userinterface, a metadata tag recommendation for a particular content item.In other embodiments, the content recommendation processing computer 301may transmit the metadata tag recommendation to a computing device, suchas client device 320, a mobile computing device, and the like. The usermay review the recommended/suggested metadata tag value, and providefeedback to the content recommendation processing system indicatingwhether the metadata tag value is correct and/or accurately reflects thesubject matter of the content item. In still another example, the usermay be prompted to identify and/or assign metadata tag values to acontent item that accurately reflect the subject matter of the available(and/or consumed) content item.

In other aspects of the present disclosure, the content recommendationprocessing computer 301 may utilize the generated feedback to adjust theapplicability score for a metadata tag value assigned to a content itemby repeating one or more of steps 601 and 603. In other aspects of thepresent disclosure, the content recommendation processing computer 301may be configured to dynamically update applicability scores for one ormore metadata tags based on feedback received at the contentrecommendation processing computer 301. For example, in the instancethat a user indicates that a first metadata tag value (e.g., “horror”)does not accurately reflect a metadata field (e.g., “genre”) of acontent item, the content recommendation processing computer 301 may beconfigured to reduce the applicability score assigned to the horrormetadata tag value for the content item by a threshold value orpercentage. For example, the content recommendation processing computer301 may reduce the applicability score assigned to the horror metadatatag value by 5 percent. In other aspects of the present disclosure, theapplicability score of a user-assigned metadata tag value for a contentitem may be adjusted by the content recommendation processing computer301 based on how many tag values have already been assigned to thecontent item. For example, in the instance that a user assigns a firstmetadata tag value (e.g., “action”) to a metadata field (e.g., “genre”)of a content item that does not having existing metadata and/or has notbeen previously annotated/associated with the particular metadata tagvalue (e.g., because the applicability score falls below a firstthreshold), the content recommendation processing computer 301 may beconfigured to increase the applicability score assigned to the actionmetadata tag value under the genre metadata field of the content item bya predetermined value. For example, the content recommendationprocessing computer 301 may increase the applicability score assigned tothe action metadata tag value by 1 percent.

As discussed above, one or more content items analyzed by the contentrecommendation processing computer 301 may have been previouslyannotated with metadata tag values, for example, a system administratormay have manually assigned metadata tag values to one or more contentitems. The content recommendation processing computer 301 may beconfigured to replace previously annotated metadata tag information fora content item using applicability scores generated for the contentitem. For example, a first content item (e.g., “Aliens on the Moon”)available to a user for consumption may have been previously annotatedwith five (5) metadata tag values (i.e., Documentary, Science Fiction,Horror, Thriller, and Comedy) for the genre metadata field of the firstcontent tem. In this example, the “Aliens on the Moon” content item hasnot been previously annotated with metadata tag values relating to othermetadata fields (e.g., Theme, Tone, Keyword, etc.). The contentrecommendation computer 301 may determine whether additional metadatamay be provided for the movie “Aliens on the Moon” by comparing themovie “Aliens on the Moon” to other content items and respectivemetadata tag values as described above in step 603 (as well as steps505-519). For example, the content recommendation processing computer301 may determine whether “Aliens on the Moon” is similar to one or moreother content items having metadata fields and/or metadata tag valuesthat have not been associated with and/or assigned to the movie “Alienson the Moon.” Accordingly, the content recommendation processingcomputer 301 may be configured to generate weight values for and assignmetadata tag values to the “Aliens on the Moon” movie.

Table 2 shown below, depicts an example set of applicability scoresgenerated by the content recommendation processing system for variousmetadata tag values associated with the movie “Aliens on the Moon”:

TABLE 2 Genre Theme Tone Keyword Science & Tech. 0.0506 Space & Aliens0.0094 Tense 0.0049 UFO 0.0100 Documentary 0.0311 Alien Encounter 0.0080Creepy 0.0045 Alien 0.0078 Space 0.0299 Evil Aliens 0.0050 Rousing0.0033 Danger 0.0068 Paranormal 0.0289 Space Travel 0.0039 Deadly 0.0031Disaster 0.0058 Science Fiction 0.0274 Save the World 0.0038 Gruesome0.0010 Chaos 0.0051 Genre: Horror 0.0169 Genre: Thriller 0.0077 Genre:Comedy 0.0020

As shown in Table 2 above, only two of the five metadata tags originallyassigned to the “Aliens on the Moon” movie (e.g., Documentary andScience Fiction) are within the top 5 applicability scores of metadatatag values within the genre metadata field. However, the contentrecommendation processing computer 301 has determined that several othermetadata tag values within the genre metadata field for the movie“Aliens on the Moon” have higher applicability scores than the metadatatags originally assigned to the movie.

As will be explained in more detail below with respect to steps 607 and609, the content recommendation processing computer 301 may beconfigured to remove one or more of the existing and/or previouslyannotated metadata tag values for the “Aliens on the Moon” movie basedon applicability score threshold. Additionally or alternatively, thecontent recommendation processing computer 301 may be configured toreplace one or more of the existing and/or previously annotated metadatatag values for the “Aliens on the Moon” with metadata tag valuesdetermined by the content recommendation processing system as bettercharacterizing (e.g., having a higher applicability score) the subjectmatter and attributes of the movie. Furthermore, as shown above in Table2, the content recommendation processing computer 301 has identified aplurality of metadata tags for the “Aliens on the Moon” content itemunder the metadata fields of Theme, Tone, and Keyword. The contentrecommendation processing computer 301 may be further configured toassign the generated metadata tags under the Theme, Tone, and Keywordmetadata fields to the Aliens on the Moon” movie.

In still other aspects of the present disclosure, the contentrecommendation processing computer 301 may calibrate and/or adjustapplicability scores for a metadata tag value assigned to a content itemby utilizing previously annotated metadata tag value information for thecontent item. For instance, the content recommendation processingcomputer 301 may calibrate metadata tag applicability scores utilizing aset of verified metadata tag values. As discussed above, utilizing usercontent consumption habits and preferences to determine a level ofsimilarity between content items may result in inaccuracies that becomecumulative over time. Such errors may be limited by conducting periodiccalibration and testing of metadata tag values generated by the computer301 for content items. Periodic calibration of the contentrecommendation processing system utilizing human-verified metadata tagvalues may aid the content recommendation processing system in resettingand/or improving the accuracy of generated metadata tag values. In someembodiments, a content provider may utilize system administrators and/orother personnel to verify metadata tag values assigned to content itemsby the content recommendation processing computer 301.

For example, a system administrator may review a plurality of metadatatag values generated by the content recommendation processing computer301 for one or more content items to confirm the appropriateness orapplicability of the metadata tag annotation. The content recommendationprocessing computer 301 may utilize feedback from the systemadministrator to remove metadata tag annotations for a content itemand/or to adjust applicability scores for a content item. As anotherexample, the system administrator may introduce and/or add new contentitems to the plurality of content items available for consumption tousers. The newly introduced content items may include (and/or have beenmanually assigned) metadata tags that accurately characterize thesubject matter of the content items. The content recommendationprocessing computer 301 may determine updated applicability scores forthe plurality of content items available for consumption to users, asdescribed above in steps 503-519. If the content recommendationprocessing computer 301 determines that an existing applicability scorefor a metadata tag value differs from the updated applicability score,the content recommendation processing computer 301 may adjust thereplace the existing applicability score with the updated score. In someembodiments, the content recommendation processing computer 301 maytransmit a notification to the system administrator indicating aproposed adjustment to (or replacement of) the applicability score forthe metadata tag value, and may be further configured to prompt thesystem administrator for feedback confirming the proposed scoreadjustment/replacement.

In other embodiments, the content recommendation processing computer 301may receive input from a user indicating a correctness (and/or accuracy)for one or more metadata tag values generated by the contentrecommendation processing computer 301 for a content item. For example,a user may provide feedback via the electronic program guide (and/orother suitable user interface) indicating that a first metadata tagvalue, in the one or more metadata tag values generated by the system,has been incorrectly assigned to a particular content item. As anotherexample, the user may provide feedback indicating a user preference thatthe first metadata tag not be associated with the particular contentitem. As yet another example, the user may provide feedback indicating aproposed or suggested weighting (e.g., applicability score) for thefirst metadata tag and the particular content item.

At step 607, the content recommendation processing computer 301 maydetermine whether an applicability score for the metadata tag value andcontent item received at step 603 satisfies a threshold value. Thecontent recommendation processing computer 301 may compare anapplicability score, such as a calibrated applicability score determinedat step 605, to a threshold value to assess the applicability and/orappropriateness of the content item being annotated with metadata tagvalue. In some instances, a computing device may push the thresholdvalue to the content recommendation processing computer 301. In otherinstances, the content recommendation processing computer 301 mayreceive from another computing device data indicating the predeterminedthreshold value in response to a request. In still other instances, thecontent recommendation processing computer 301 may receive input from asystem administrator, via an input device, identifying the predeterminedthreshold value. Various threshold values may be utilized by the contentrecommendation processing computer 301 when determining whether tomaintain and/or remove a metadata tag value from a metadata field of acontent item without departing from the scope of the present disclosure.

At step 607, if the applicability score for the metadata tag value andcontent item does not satisfy the threshold value, the method mayproceed to step 609 where the content recommendation processing computer301 may remove the metadata tag value from a metadata field of thecontent item. Referring back to the example in Table 2 above, whendetermining whether to remove and/or maintain a particular metadata tagvalue for the genre metadata field of the movie “Aliens on the Moon,”the content recommendation processing computer 301 may compare theapplicability score for the particular metadata tag value to apredetermined threshold value, such as 0.0150. In this example, thecontent recommendation processing computer 301 may compare theapplicability score for the Science Fiction metadata tag value (i.e.,0.0274) to the threshold applicability score (i.e., 0.0150) to determinewhether the Science Fiction metadata tag value is appropriate for themovie “Aliens on the Moon.” In this example, the content recommendationprocessing computer 301 would determine that the Science Fictionmetadata tag satisfies the threshold value, and thus, will maintain thisparticular metadata tag annotation for the movie. As another example,the content recommendation processing computer 301 may compare theapplicability score for the Horror metadata tag value (i.e., 0.0077) tothe threshold applicability score (i.e., 0.0150) to determine the Horrormetadata tag value is not an appropriate metadata tag annotation for themovie “Aliens on the Moon.” Accordingly, the content recommendationsystem may remove the Horror metadata tag value from the genre metadatafield for the movie “Aliens on the Moon.”

In some aspects of the present disclosure the content recommendationprocessing system may track the various metadata tag values that areremoved from content items for a particular user over time. The contentrecommendation processing computer 301 may generate a history ofmetadata tag removals by storing in memory each instance of a metadatatag being removed from a content item. The content recommendationprocessing computer 301 may associate the removed metadata tag with thecorresponding content item, and/or otherwise identify the correspondingcontent item from which the metadata tag was removed. The contentrecommendation processing computer 301 may utilize this history ofmetadata tag removals along with a content consumption history of theuser to generate and our modify user content recommendations andpreferences. For example, the content recommendation processing computer301 may determine that several movies previously consumed by the userhave had a particular metadata tag (e.g., horror metadata tag) removedwithin a recent time period (e.g., the last week, month, three months,etc.). Additionally or alternatively, the content recommendationprocessing computer 301 may analyze previously consumed content itemsfor which the user has indicated a content preference, such as byproviding a high ranking, a thumbs-up, or any other suitable preferenceindication, when determining which content items have had a metadata tagremoved.

In some embodiments, the content recommendation processing computer 301may transmit a communication to the user indicating that the horrormetadata tag has been removed from a threshold number of content items.Additionally or alternatively, the content recommendation processingcomputer 301 may prompt the user to adjust user content preferences,such as reducing the relevance of horror movies when recommendingcontent items, given that a threshold number of horror metadata tagswere recently removed from content items consumed by the user. In someinstances, the content recommendation processing computer 301 mayautomatically adjust user content preferences based on the history ofmetadata tag removals. In some embodiments, the content recommendationprocessing computer 301 may adjust and/or calibrate contentrecommendations to the user based on the history of metadata tagremovals. Referring to the example above, when determining a contentrecommendation for the user, the content recommendation processingcomputer 301 may reduce the relevance of those content items annotatedwith horror metadata tag values comprising high applicability scores(e.g., applicability scores satisfying a score threshold). For example,as will be discussed in more detail below, the content recommendationprocessing computer 301 may reduce a composite recommendation value forone or more content items with horror metadata tag values satisfying athreshold applicability score.

In other aspects of the present disclosure, as discussed above, thecontent recommendation processing system may replace the horror metadatatag value with another metadata tag value that has yet to be annotatedto the movie “Aliens on the Moon.” Referring to the example above, thecontent recommendation processing computer 301 may identify a metadatatag value that has yet to be annotated to the movie “Aliens on theMoon,” such as the Paranormal metadata tag value, and may compare theapplicability score for the particular metadata tag value to thepredetermined threshold value. In this example, the contentrecommendation processing computer 301 may compare the applicabilityscore for the Paranormal metadata tag value (i.e., 0.0289) to thethreshold applicability score (i.e., 0.0150) to determine whether theParanormal metadata tag value is appropriate for the movie “Aliens onthe Moon.” In this example, the content recommendation processingcomputer 301 would determine that the Paranormal metadata tag satisfiesthe threshold value, and thus, may assign this particular metadata tagvalue to the genre metadata field of the movie. Additionally oralternatively, the content recommendation processing system may modifythe genre metadata field of the movie “Aliens on the Moon” to replacethe Horror metadata tag value, which fails to satisfy the applicabilityscore threshold value, with the Paranormal metadata tag value. Thecontent recommendation processing computer 301 may transmit an alert orother suitable communication to a computing device indicating anymodifications to a metadata field or metadata tag value for a contentitem. Referring to the examples above, the content recommendationprocessing computer 301 may transmit a communication to a systemadministrator indicating that the Science Fiction metadata tag value hasbeen assigned to the genre metadata field for the movie “Aliens on theMoon.” As another example, the content recommendation processingcomputer 301 may transmit a communication to a user indicating that theHorror metadata tag value can been removed from (and/or disassociatedwith) the genre metadata field for the movie “Aliens on the Moon.”Additionally or alternatively, the content recommendation processingcomputer 301 may transmit a communication to the user indicating thatthe Paranormal metadata tag value has or may replace the Horror metadatatag value in the genre metadata field for the movie “Aliens on theMoon.”

In still other aspects of the present disclosure, as discussed above,the content recommendation processing computer 301 may receive feedbackfrom a user indicating whether a metadata tag annotation for a contentitem is accurate and/or appropriate. In some instances, the contentrecommendation processing computer 301 may prompt the user for feedbackwhen transmitting an alert or communication indicating a change to ametadata tag value for a content item. Referring to the example above,the content recommendation processing computer 301 may prompt the userto provide feedback as to the appropriateness of removing the Horrormetadata tag value for the movie “Aliens on the Moon” when transmittinga notification indicating a proposed change to the Horror metadata tagvalue for the movie. As an example, the notification indicating aproposed change to Horror metadata tag value may prompt the user toprovide feedback, such as by prompting the user to confirm whether theHorror metadata tag value should be removed. In some instances, thecontent recommendation processing computer 301 may prompt the user forfeedback before modifying the metadata field of a content item to removea metadata tag value. After receiving the user feedback, the contentrecommendation processing computer 301 may calibrate the applicabilityscore for one or more metadata tag values, as described above in step605. The content recommendation processing computer 301 may beconfigured to perform one or more of steps 605-609 after receiving theuser feedback. Referring to the example above, the contentrecommendation processing computer 301 calibrate the applicability scorefor the Horror metadata tag and may determine whether the calibratedscore satisfies the threshold value, as described in step 607.

As discussed above with respect to FIGS. 4B, 5 and 6, the contentrecommendation processing computer 301 may be configured to generate anapplicability score for a metadata tag value and a content itemindicating a degree of suitability for annotating a content item with ametadata tag value. In additional aspects of the present disclosure, thecontent recommendation processing computer 301 may be configured togenerate content recommendations based at least in part on applicabilityscores determined for metadata tag values of various content items thatare available for consumption to users.

FIG. 7 shows an exemplary flowchart of a process for generating usercontent recommendations and calibrating metadata tag values for acontent recommendation processing computer 301 according to one or moreillustrative aspects of the disclosure. As discussed above, a user mayutilize an electronic program guide and/or other suitable user interfaceto browse and consume content items. A user may query the electronicprogram guide for content recommendations, such as requesting a list ofhorror movies that are available for consumption, and/or requesting alist of content items that are similar to a particular movie. Thecontent recommendation processing computer 301 may be configured toutilize applicability scores for a metadata tag value and content itemto more efficiently and accurately determine the appropriate contentitems to present to a user in response to a content query. Further, asmore content items are made available to users for consumption and asusers provide feedback on the their content preferences and the accuracyof assigned applicability scores, an improved content recommendationcomputer 301 configured to generate and provide content recommendationsin accordance with the determined applicability scores is disclosedherein.

The steps shown in FIG. 7 may be performed by a content recommendationprocessing system (e.g., by content recommendation processing system300). For example, the steps may be implemented by contentrecommendation processing computer 301. The steps may also beimplemented by a distributed computing system, having devices at variouslocations. One or more computing devices may be implemented as acomputing device using the structure shown in FIG. 2, and may beconfigured to respond to user requests for content, such as streamingvideo or audio. For brevity, the following description will generallyassume that the steps shown in FIG. 7 are performed by contentrecommendation processing computer 301.

In step 701, the content recommendation processing computer 301 may beconfigured to receive a content request/query. The contentrecommendation processing computer 301 may receive a content request orquery from a computing device (e.g., client device 320). A user mayutilize a user interface, such as an electronic program guide, to browsecontent items and to initiate a content request or query for contentitems. For example, a user may request the user interface to display alisting of horror movies that are available for consumption. In thisexample, the content recommendation processing computer 301 may receive,from the client device 320, a content request for all horror movies thatare available for the user to consume. As another example, a user mayrequest the user interface to display a listing of movies that aresimilar to the movie Shrek. The content recommendation processingcomputer 301 may receive, from the client device 320, a content requestfor a listing of available movies that are similar to the movie Shrek.As discussed above, there are a variety of ways in which the contentrecommendation processing computer 301 may determine that two contentitems are similar. For example, a similarity between two content itemsmay be express by the ratio of the number of users that like a firstitem and a second item to the expected number of users that like thefirst item and the second item. Additionally or alternatively, a levelof similarity between two content items may be determined and/oradjusted based on a comparison of metadata fields, metadata tag valuesand corresponding applicability scores between the two content items.

In step 702, the content recommendation processing computer 301 may beconfigured to retrieve one or more applicability scores in response tothe content request received at step 701. The content recommendationprocessing computer 301 may analyze the content request received at step701 and may be configured to identify one or more content itemsresponsive to the content request. For example, in response to a userrequest to view a listing of action movies, the content recommendationprocessing computer 301 may identify a plurality of content items thatinclude an action metadata tag value. As another example, in response toa user request to view a listing of content items that are similar to aparticular movie (e.g., a target content item), the contentrecommendation processing computer 301 may analyze the target item todetermine the one or more metadata fields and tag values that define thetarget content item, and may identify a plurality of content items thatinclude the same metadata fields and/or metadata tag values as thetarget content item. The content recommendation processing computer 301may retrieve applicability scores of metadata tag values for one or morecontent items responsive to the user's content request. The contentrecommendation processing computer 301 may retrieve/generateapplicability scores of metadata tag values in a similar manner as steps517 and 603.

At step 703, the content recommendation processing computer 301 may rankand/or sort metadata tag values for one or more content items based onapplicability scores. As discussed above, an applicability scoregenerated for a metadata tag value and content item may indicate alevel/degree of similarity between the content item and a plurality ofother content items that have been annotated with the particularmetadata tag value. In some aspects of the present disclosure, thecontent recommendation processing computer 301 may rank and/or sort aplurality of metadata tag values by applicability score for a particularmetadata field of a content item (e.g., genre, theme, tone, etc).Accordingly, by ranking metadata tag values of content items byapplicability score, the content recommendation processing computer 301may identify those content items whose subject matter and/or contentcharacteristics best represent/embody a particular metadata tag value.For example, a movie having the highest applicability score for a horrormetadata tag value may indicate that his movie best represents/embodiesthe characteristics and attributes of the prototypical horror movie.

Additionally or alternatively, the content recommendation processingcomputer 301 may be configured to rank/sort content items based onrespective metadata tag values and corresponding applicability score. Asdiscussed above, a user may query a user interface to identify contentitems similar to a target content item. In such instances, the contentrecommendation processing computer 301 may analyze the metadata fieldsand tag values of the target content item to identify other contentitems available for consumption that comprise similar metadata fieldsand metadata tag values. In some aspects of the present disclosure, foreach metadata tag value within a metadata field of the target contentitem, the content recommendation processing computer 301 may beconfigured to rank/sort the identified content items by applicabilityscore that most closely approximates (and/or has the smallest deviationfrom) the applicability score of the metadata tag value for the targetcontent item.

The first column of Table 3 (below) shows an example listing of contentitems (and respective applicability score) that have been identified bycontent recommendation computer 301 as comprising similar metadatafields (i.e., “genre”) and metadata tag values (e.g., action and comedy)as a target content item, the action movie “Total Recall.”

TABLE 3 Target Content Item: Total Recall Weighting Score for ActionTag: 0.5200 Weighting Score for Comedy Tag: 0.5000 Action Tag ContentItem Action Tag Content Item Comedy Tag Content Item Weight (sorted)Weight (sorted) Weight Miami Vice 0.5600 Interstellar 0.5250 Anchorman0.7600 Beverly Hills Cop 0.5500 Judgment Day 0.5100 Pineapple Express0.6500 Transformers 0.5400 Transformers 0.5400 Old School 0.4600Interstellar 0.5250 Beverly Hills Cop 0.5500 Family Vacation 0.4500Judgment Day 0.5100 Miami Vice 0.5600 Beverly Hills Cop 0.4000 Shrek0.4100 Shrek 0.4100 Zoolander 0.3100 Pineapple Express 0.1000 PineappleExpress 0.1000 Shrek 0.2100

In the event that the movie “Total Recall” has an action metadata tagvalue having an applicability score of 0.5200, the contentrecommendation processing computer 301 may rank/sort the identifiedcontent items by respective applicability scores of the action metadatatag value that are closest to the value of 0.5200. As shown in column 2of Table 3, the movie Interstellar is ranked first as it has anapplicability score of the action metadata tag value (i.e., 0.5250),which is closest to the applicability score of the action metadata tagvalue of the movie Total Recall Judgment Day is ranked second as itsrespective applicability score of the action metadata tag value onlydeviates from that of Total Recall by 0.0100, while the movieTransformers is ranked third because its respective applicability scoreof the action metadata tag value deviates from that of Total Recall by0.0200. As will be explained in more detail below, the contentrecommendation processing computer 301 may utilize the determinedranking/sorting of content items in view of the metadata fields and tagvalues of a target content item to generate user contentrecommendations.

At step 704, the content recommendation processing computer 301 maygenerate and/or transmit content recommendations to a user in view ofthe content request received at step 701. For example, if a userrequested a listing of action movies available for consumption, thecontent recommendation computer 301 may identify those content itemshaving the highest applicability scores for the action metadata tagvalue in view of the rankings/sorting determined at step 703, and maytransmit the identified content items to the user. In some instances,the content recommendation processing computer 301 may identify athreshold number of content items having the highest applicabilityscores for a particular metadata tag value in view of therankings/sorting determined at step 703.

As another example, if a user requested a listing of content items thatare similar to a target item, the content recommendation computer 301may identify those content items having metadata tag values and/orapplicability scores that most closely correspond to the metadata tagvalues and/or applicability scores of the target item. Referring back toTable 3 above, in this example, in response to a user content requestfor more content items like the movie Total Recall (e.g., a targetcontent item), the content recommendation processing computer 301 mayrecommend the movie Interstellar to the user. In some aspects of thepresent disclosure, the content recommendation processing computer 301may analyze a plurality of metadata tag values and/or metadata fieldsfor a target item to determine a content recommendation for the user.Referring to the example in Table 3 above, the second and third columnsof the table identify content items that have applicability scores foraction and comedy metadata tag values that are similar to theapplicability scores for similar metadata tag values assigned to themovie “Total Recall,” and the content items have been sorted byrespective applicability scores that most closely approximate thecorresponding applicability scores of the movie “Total Recall.” For eachmetadata tag value associated with the target item (e.g., “TotalRecall”), the content recommendation processing computer 301 may assigna recommendation score to the related content items based on theirrespective applicability score rank.

Referring to the example in Table 3, for content items in column 2 ofthe Table having applicability scores similar to Total Recall withrespect to the action metadata tag, the content recommendationprocessing computer 301 may assign recommendation scores to the contentitems in their respective applicability score ranking. For example, thecontent recommendation processing computer 301 may assign arecommendation score of 100 to the movie Interstellar; 90 to JudgmentDay; 80 to Transformers; 70 to Beverly Hills; 60 to Miami Vice and soforth. Similarly, for content items in column 3 of the Table havingapplicability scores similar to Total Recall for the comedy metadatatag, the content recommendation processing computer 301 may assign arecommendation score of 100 to the movie Old School; 90 to FamilyVacation; 80 to Beverly Hills; 70 to Pineapple Express; 60 to Zoolanderand so forth. The content recommendation processing computer 301 maygenerate a composite recommendation score for each content item based ona combination (and/or sum) of the respective recommendation scoresdetermined for each metadata tag value and/or metadata field.

In the above example, the content recommendation processing computer 301may determine the composite recommendation score for the movie BeverlyHills Cop by computing the average of the individual recommendationscores for the action and comedy metadata tag values, resulting in acomposite recommendation score of 70 (i.e., (60+80)/2). The contentrecommendation processing computer 301 may be configured to recommend toa user the one or more content items having the highest compositerecommendation score. Based on the example in Table 3, in response to auser requesting to view additional content items like the movie TotalRecall, the content recommendation processing computer 301 may recommendto the user one or more content items having the highest compositerecommendation scores, such as Beverly Hills Cop (65), Pineapple Express(55), and Shrek (45). Additionally or alternatively, the contentrecommendation processing system may be configured to receive and storeuser content preferences. A user may indicate a content preference byidentifying one or more particular metadata fields and/or tag valuesthat are most desirable to the user. The content recommendationprocessing computer 301 may be configured to adjust contentrecommendations based on user content preferences. For example, the usermay indicate a preference for action and drama movies. In someembodiments, the content recommendation processing computer 301 may beconfigured to weight the recommendation score assigned to content itemshaving metadata tag values that are preferred by the user.

In some aspects of the present disclosure, a user may browse a list ofmovie genres in an electronic program guide and may select an “action”menu to view a list of action movies available for consumption. Inresponse to the user selection, the content recommendation processingsystem may display a listing of content items having the highestapplicability scores for the action metadata tag. In some embodiments,the content recommendation processing computer 301 may identify one ormore content items having the highest applicability scores for aparticular metadata tag value as a favorite content item. The contentrecommendation processing computer 301 may utilize informationindicating a favorite content item for a user to modify the display ofcontent items and other information displayed in the user's electronicprogram guide. For example, the content recommendation processing systemmay visually emphasize (e.g., highlight, outline, bold, etc.) thosecontent items that have been identified as a favorite content item forthe user.

As discussed above, the content recommendation processing computer 301may identify one or more metadata fields that comprise key indicators ofuser content preferences. For example, the content recommendationprocessing computer 301 may analyze user content preferences todetermine that a user prefers comedy and horror movies more than actionmovies. As such, the content recommendation processing computer 301 mayrecommend content items to the user corresponding to those content itemshaving the highest applicability scores for the comedy and horrormetadata tag values. Similarly, the content recommendation processingcomputer 301 may recommend to a user those content items having a lowapplicability score for the action metadata tag value. Additionally oralternatively, the content recommendation processing computer 301 maydeprioritize and/or suppress content recommendations transmitted to auser for those content items having high applicability scores for theaction metadata tag value, and may be further configured to suppressand/or deprioritize content recommendations transmitted to a user forcontent items having low applicability scores for comedy and horrormetadata tag values.

In still other aspects of the present disclosure, the contentrecommendation processing computer 301 my utilize metadata tag rankingsdetermined at step 703 to affect the presented order and/or context ofcontent items presented in a program guide. Referring to the example inTable 1 above, a first content item (e.g., “Shrek”) may be annotatedwith both a “Children Entertainment” metadata tag value and an “Action”metadata tag value under the genre metadata field. An electronic programguide and/or other suitable user interface for accessing and consumingcontent may provide the user with an ability to search for content itemscorresponding to a particular metadata field of a content item (e.g.,searching content based on genre, theme, keywords, etc.). For example, auser may search for content items that are “Action” movies. Theelectronic program guide may provide a listing of content items thatfall under the category of “Action” movies. When displaying theresulting content items, a user may anticipate that content itemspresented toward the top of the results are most closely associated withthe selected genre (e.g., action movies).

Although the movie Shrek has been annotated/tagged as an action movie,it may not be appropriate to display Shrek toward the top of the listingof action movies in the user interface. Accordingly, as discussed above,an applicability score for a particular metadata tag value and contentitem, as determined by the content recommendation processing computer301, may be utilized to determine to what degree a content item isembodied/characterized by the particular metadata tag value, such thatthe content recommendation processing computer 301 (and/or a contentprovider) may adjust the ranking/order of content items when presentingthe results of a user content request within the user interface. Forexample, the content recommendation processing computer 301 may identifythe one or more content items having the highest applicability scoresfor a first metadata tag value (e.g., horror) under the genre metadatafield, and may modify the presentation/display of content items withinthe user interface to prioritize/emphasize the one or more identifiedcontent items in response to a search query, such as a query for horrormovies.

Although example embodiments are described above, the various featuresand steps may be combined, divided, omitted, rearranged, revised and/oraugmented in any desired manner, depending on the specific outcomeand/or application. Various alterations, modifications, and improvementswill readily occur to those skilled in art. Such alterations,modifications, and improvements as are made obvious by this disclosureare intended to be part of this description though not expressly statedherein, and are intended to be within the spirit and scope of thedisclosure. Accordingly, the foregoing description is by way of exampleonly, and not limiting. This patent is limited only as defined in thefollowing claims and equivalents thereto.

We claim:
 1. A method, implemented by one or more computing devices,comprising: receiving a first metadata tag associated with a firstcontent item; determining a first similarity data indicating a level ofsimilarity between the first content item and a plurality of othercontent items; determining, based on the first similarity data, a firstset of content items, of the plurality of other content items,satisfying a threshold level of similarity to the first content item;determining a second set of content items, of the plurality of othercontent items, associated with the first metadata tag; determining athird set of content items comprising one or more content items in thefirst set of content items that are associated with the first metadatatag; generating, by a processor, an applicability score for the firstmetadata tag and the first content item based on: (i) a secondsimilarity data indicating a level of similarity between the first setof content items and the third set of content items, and (ii) a ratio ofa number content items in the third set of content items to a number ofcontent items in the second set of content items; and transmitting arecommendation of a second content item based on the applicabilityscore.
 2. The method of claim 1, wherein the first similarity datacomprises at least a first similarity value indicating a level ofsimilarity between the first content item and another content item inthe first set of content items.
 3. The method of claim 1, wherein thesecond similarity data comprises at least a first similarity valueindicating a level of similarity between the first content item and acontent item in the second set of content items.
 4. The method of claim1, wherein the first similarity data comprises a plurality of similarityvalues, and wherein each similarity value indicates a level ofsimilarity between the first content item and a content item in theplurality of other content items.
 5. The method of claim 4, furthercomprising: determining a total number of users; determining a firstproportion indicating a proportion of the total number of users thathave consumed the first content item and the content item in theplurality of other content items; and determining a second proportionindicating a proportion of the total number of users that have consumedthe first content item.
 6. The method of claim 5, wherein a firstsimilarity value indicating a level of similarity between the contentitem in the plurality of other content items and the first content itemcomprises a ratio of the first proportion to the second proportion. 7.The method of claim 5, wherein a first similarity value indicating alevel of similarity between the content item in the plurality of othercontent items and the first content item comprises a difference betweenthe first proportion and the second proportion.
 8. A method, implementedby one or more computing devices, comprising: receiving input indicatinga first metadata tag and a first content item; determining, for aplurality of other content items, a corresponding similarity valueindicating a level of similarity between the first content item and theother content item; determining a first set of content items, of theplurality of other content items, comprising similarity values thatsatisfy a threshold; determining a second set of content items, of theplurality of other content items, associated with the first metadatatag; determining a first ratio of a sum of similarity values for thefirst set of content items to a sum of similarity values for the secondset of content items; determining a second ratio of a number of contentitems in the first set of content items to a number of content items inthe second set of content items; generating an applicability score forthe first metadata tag and the first content item based at least in parton the first ratio and the second ratio; and transmitting arecommendation for the first content item based on the applicabilityscore generated for the first metadata tag.
 9. The method of claim 8,further comprising: assigning the applicability score to the firstmetadata tag.
 10. The method of claim 9, wherein the assigning theapplicability score to the first metadata tag further comprises:determining that that the applicability score satisfies a first minimumthreshold applicability score.
 11. The method of claim 8, furthercomprising: determining that the applicability score satisfies a secondminimum threshold applicability score; and in response to thedetermining that the applicability score satisfies the second minimumthreshold applicability score, assigning the first metadata tag to ametadata field of the first content item.
 12. The method of claim 11,wherein the assigning the first metadata tag further comprises:determining that the metadata field of the first content item is notassociated with a metadata tag; and in response to assigning the firstmetadata tag to the metadata field, increasing the applicability scoreby a predetermined value.
 13. The method of claim 8, further comprising:ranking, by the applicability score, a plurality of metadata tags for afirst metadata field of the first content item; determining a first setof metadata tags, of the plurality of metadata tags, comprising thehighest applicability scores; and assigning one or more metadata tags inthe first set of metadata tags to the first metadata field of the firstcontent item.
 14. A method, implemented by one or more computingdevices, comprising: receiving a first input indicating a first contentitem; receiving a second input indicating a threshold applicabilityscore; receiving a first set of metadata tags associated with the firstcontent item; determining applicability scores for the first set ofmetadata tags; determining that an applicability score for a firstmetadata tag, of the first set of metadata tags, fails to satisfy thethreshold applicability score; receiving user feedback data relating tothe first metadata tag of the first set of metadata tags and the firstcontent item; adjusting the applicability score for the first metadatatag, of the first set of metadata tags, based on the received feedbackdata; comparing the adjusted applicability score for the first metadatatag, of the first set of metadata tags, to the threshold applicabilityscore; and modifying a metadata field of the first content item toremove the first metadata tag based on the comparing the adjustedapplicability score to the threshold applicability score.
 15. The methodof claim 14, further comprising: determining a plurality of othermetadata tags associated with the first content item; determining asecond metadata tag, of the plurality of other metadata tags, comprisingan applicability score satisfying the threshold applicability score; andmodifying the metadata field of the first content item to replace thefirst metadata tag of the first set of metadata tags with the secondmetadata tag of the plurality of other metadata tags.
 16. The method ofclaim 14, wherein each metadata tag in the first set of metadata tagshas been assigned to at least a threshold number of other content items.17. The method of claim 16, wherein the threshold number of othercontent items comprises a number of content items within a range of fourcontent items to sixteen content items.
 18. The method of claim 16,further comprising: receiving, via an input device, a third inputindicating a content query; determining a second metadata tag, whereinthe second metadata tag is associated with the content query;determining a plurality of content items associated with the secondmetadata tag; determining applicability scores for one or more contentitems in the plurality of content items; and determining, byapplicability score, rankings for the plurality of content itemsassociated with the second metadata tag.
 19. The method of claim 18,further comprising: in response to receiving the content query,transmitting a content recommendation based on the determined rankingsfor the plurality of content items.
 20. The method of claim 14, furthercomprising: determining, by a processor, a history of metadata tagremoval for one or more content items; adjusting, based on the historyof metadata tag removal, a recommendation value for the one or morecontent items; and transmitting a content recommendation for a secondcontent item based on the recommendation value.